THE SEMANTIC WEB CRAFTING INFRASTRUCTURE FOR AGENCY
Bo Leuf Technology Analyst, Sweden
THE SEMANTIC WEB
THE SEMANTIC WEB CRAFTING INFRASTRUCTURE FOR AGENCY
Bo Leuf Technology Analyst, Sweden
Copyright # 2006
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) 1243 779777
Email (for orders and customer service enquiries):
[email protected] Visit our Home Page on www.wiley.com All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to
[email protected], or faxed to (+44) 1243 770620. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany John Wiley & Sons Australia Ltd, 42 McDougall Street, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop # 02-01, Jin Xing Distripark, Singapore 129809 John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging-in-Publication Data Leuf, Bo, Technology Analyst, Sweden. The Semantic Web: crafting infrastructure for agency/Bo Leuf. p. cm. Includes bibliographical references and index. ISBN 0-470-01522-5 1. Semantic Web. I. Title. TK5105.88815.L48 2005 025.04–dc22 2005024855 British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN-13 978-0-470-01522-3 (HB) ISBN-10 0-470-01522-5 (HB) Typeset in 10/12pt Times Roman by Thomson Press (India) Limited, New Delhi Printed and bound in Great Britain by Antony Rowe, Chippenham, Wilts This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production.
I dedicate this book to the many visionaries, architects, and programmers, who all made the Internet an interesting place to ‘live’ and work in. And especially to Therese.
Contents Foreword Preface Part I
xiii xv
Content Concepts
1
Enhancing the Web Chapter 1 at a Glance There and Back Again Resource Identifiers Extending Web Functionality From Flat Hyperlink Model To Richer Informational Structures The Collaboration Aspect Extending the Content Model Mapping the Infosphere Well-Defined Semantic Models
3 3 4 5 11 17 22 23 26 27 30
2
Defining the Semantic Web Chapter 2 at a Glance From Model to Reality The Semantic Web Concept Representational Models The Road Map Identity Markup Relationships Reasoning Agency on the Web Semantic P2P The Visual Tour The Map The Architectural Goals The Implementation Levels
31 31 32 33 43 46 46 49 49 51 52 52 53 54 55 57
viii
Contents
3
Web Information Management Chapter 3 at a Glance The Personal View Creating and Using Content Authoring Publishing Exporting Databases Distribution Searching and Sifting the Data Semantic Web Services Security and Trust Issues XML Security Trust A Commercial or Free Web
4
Semantic Web Collaboration and Agency Chapter 4 at a Glance Back to Basics The Return of P2P WebDAV Peer Aspects Peering the Data Peering the Services Edge Computing Automation and Agency Kinds of Agents Multi-Agent Systems
Part II 5
59 59 60 63 64 65 66 66 69 71 81 83 84 91 97 97 98 99 101 103 103 105 106 108 109 111
Current Technology Overview
Languages and Protocols Chapter 5 at a Glance Markup The Bare Bones (HTML) Disentangling the Logic (XHTML and XML) Emerging Special-Purpose Markup Higher-Level Protocols A Messaging Framework (SOAP versus REST) Expressing Assertions (RDF) The Graphical Approach Exchanging Metadata (XMI) Applying Semantic Rules RDF Query Languages Multi-Agent Protocols
117 117 118 119 119 122 124 125 131 141 144 146 148 153
Contents
ix
6
Ontologies and the Semantic Web Chapter 6 at a Glance Ontology Defined Ontologies for the Web Ontology Types Building Ontologies Web Ontology Language, OWL Other Web Ontology Efforts Knowledge Representation Conceptual Graphs Promoting Topic Maps Description Logics
155 155 156 159 160 161 165 169 172 172 173 176
7
Organizations and Projects Chapter 7 at a Glance Major Players W3C Semantic Web Communities Dublin Core Metadata Initiative DARPA EU-Chartered Initiatives AIC at SRI Bioscience Communities Biopathways Consortium
177 177 178 178 182 183 186 187 190 193 193
8
Application and Tools Chapter 8 at a Glance Web Annotation The Annotea Project Evaluating Web Annotation Infrastructure Development Develop and Deploy an RDF Infrastructure Building Ontologies Information Management Haystack Digital Libraries Applying RDF Query Solutions Project Harmony DSpace Simile Web Syndication RSS and Other Content Aggregators Metadata Tools Browsing and Authoring Tools Metadata Gathering Tools
195 195 196 198 205 208 208 209 213 213 214 216 216 219 223 224 225 226 226 226
x
9
Contents
Examples of Deployed Systems Chapter 9 at a Glance Application Examples Retsina Semantic Web Calendar Agent MusicBrainz and Freedb Semantic Portals and Syndication SUMO Ontology Open Directory Project Ontolingua Industry Adoption Adobe XMP Sun Global Knowledge Engineering (GKE) Implemented Web Agents Agent Environments Intelligent Agent Platforms
Part III
229 229 230 232 233 235 242 245 246 251 252 255 256 256 258
Future Potential
10 The Next Steps Chapter 10 at a Glance What Would It Be Like? Success on the Web Medical Monitoring Smart Maintenance And So It Begins Meta-Critique Where Are We Now? Intellectual Property Issues The Road Goes Ever On Reusable Ontologies Device Independence
263 263 264 264 270 272 272 273 283 291 293 294 296
11 Extending the Concept Chapter 11 at a Glance Externalizing from Virtual to Physical The Personal Touch Pervasive Connectivity User Interaction Engineering Automation Adaptability Whither the Web? Evolving Human Knowledge Towards an Intelligent Web The Global Brain Conclusions Standards and Compliance
299 299 300 301 302 306 310 312 313 314 315 316 316
xi
Contents
The Case for the Semantic Web We can Choose the Benefits Part IV
318 320
Appendix Material
Appendix A Technical Terms and References At a Glance Glossary RDF RDF Schema Example Listing RDF How-to
325 325 325 333 333 336
Appendix B Semantic Web Resources At a Glance Further Reading Book Resources Other Publications Web Resources Miscellaneous
339 339 339 339 340 341 343
Appendix C Lists
345
Index
355
Foreword As an individual, as a technologist, as a business person, and as a civic participant, you should be concerned with how the Semantic Web is going to change the way our knowledgebased society functions. An encouraging sign is that one of the first large-scale community-based data pools, the Wikipedia, has grown to well over half a million articles. It is an ominous indicator that one of the first large-scale governmental metadata assignment projects is going on in China for the purpose restricting personal access to political information. The Semantic Web is not a single technology; rather it is a cluster of technologies, techniques, protocols, and processes. As computational power becomes more powerful and more ubiquitous, the amount of control that information technology will hold over people’s lives will become more pervasive, and the individual’s personal control ever less. At the same time, the employment of anonymous intelligent agents may buy individuals a new measure of privacy. The Semantic Web is the arena in which these struggles will be played out. The World Wide Web profoundly transformed the way people gain access to information; the Semantic Web will equally profoundly change the way machines access information. This change will transform yet again our own roles as creators, consumers and manipulators of knowledge. —Mitchel Ahren, Director of Marketing Operations, AdTools | Digital Marketing Concepts, Inc.
Preface This is a book that could fall into several different reader categories – popular, academic, technical – with perhaps delusional ambitions of being both an overview and a detailed study of emerging technology. The idea of writing The Semantic Web grew out of my previous two books, The Wiki Way (Addison-Wesley, 2001) and Peer to Peer (Addison-Wesley, 2002). It seemed a natural progression, going from open co-authoring on the Web to open peersharing and communication, and then onto the next version of the Web, involving peercollaboration between both software agents and human users. Started in 2002, this book had a longer and far more difficult development process than the previous two. Quite honestly, there were moments when I despaired of its completion and publication. The delay of publication until 2005, however, did bring some advantages, mainly in being able to incorporate much revised material that otherwise would have been waiting for a second edition. The broader ‘Semantic Web’ as a subject still remains more of a grand vision than an established reality. Technology developments in the field are both rapid and unpredictable, subject to many whims of fate and fickle budget allocations. The field is also ‘messy’ with many diverging views on what it encompasses. I often felt like the intrepid explorer of previous centuries, swinging a machete to carve a path through thick unruly undergrowth, pencil-marking a rough map of where I thought I was in relation to distant shorelines and major landmarks. My overriding concern when tackling the subject of bleeding-edge technologies can be summed up as providing answers to two fundamental reader questions: What does this mean? Why should I care? To anticipate the detailed exposition of the latter answer, my general answer is that you – we – should care, because these technologies not only can, but most assuredly will, affect us more than we can possibly imagine today.
Purpose and Target The threefold purpose of the book is rather simple, perhaps even simplistic: Introduce an arcane subject comprehensively to the uninitiated. Provide a solid treatment of the current ‘state of the art’ for the technologies involved. Outline the overall ‘vision of the future’ for the new Web.
xvi
Preface
My guiding ambition was to provide a single volume replete with historical background, state of the art, and vision. Intended to be both informative and entertaining, the approach melds practical information and hints with in-depth analysis. The mix includes conceptual overviews, philosophical reflection, and contextual material from professionals in the field – in short, all things interesting. It includes the broad strokes for introducing casual readers to ontologies and automated processing of semantics (not a unique approach, to be sure), but also covers a sampling of the real-world implementations and works-in-progress. However, the subject matter did not easily lend itself to simple outlines or linear progressions, so I fear the result may be perceived as somewhat rambling. Well, that’s part of the journey at this stage. Yet with the help of astute technical reviewers and the extended period of preparation, I was able to refine the map and sharpen the focus considerably. I could thus better triangulate the book’s position on the conceptual maps of both the experts and the interested professionals. The technologies described in this book will define the next generation Internet and Web. They may conceivably define much of your future life and lifestyle as well, just as the present day Web has become central to the daily activities of many – the author included. Therefore, it seems fitting also to contemplate the broader implications of the technologies, both for personal convenience and as instigator or mediator of social change. These technologies can affect us not only by the decisions to implement and deploy them, but sometimes even more in the event of a decision not to use them. Either way, the decision taken must be an informed one, ideally anchored in a broad public awareness and understanding of what the decision is about and with some insight into what the consequences might be. Even a formal go-ahead decision is not sufficient in itself. The end result is shaped significantly by general social acceptance and expectations, and it may even be rejected by the intended users. Some readers might find the outlined prospects more alarming than enticing – that is perhaps as it should be. As with many new technologies, the end result depends significantly on social and political decisions that for perspective require not just a clear vision but perhaps also a glimpse of some dangers lurking along the way. We can distinguish several categories of presumptive readers: The casual reader looking for an introduction and overview, who can glean enough information to set the major technologies in their proper relationships and thus catch a reflection of the vision. The ‘senior management’ types looking for buzzwords and ‘the next big thing’ explained in sufficient detail to grasp, yet not in such unrelenting technical depth as to daze. The industry professional, such as a manager or the person responsible for technology, who needs to get up to speed on what is happening in the field. Typically, the professional wants both general technology overviews and implementation guides in order to make informed decisions. The student in academic settings who studies the design and implementation of the core technologies and the related tools.
Preface
xvii
The overall style and structure of the book is held mainly at a moderate level of technical difficulty. On the other hand, core chapters are technical and detailed enough to be used as a course textbook. All techno-jargon terms are explained early on. In The Semantic Web, therefore, you are invited to a guided journey through the often arcane realms of Web technology. The narrative starts with the big picture, a landscape as if seen from a soaring plane. We then circle areas of specialist study, thermal-generating ‘hotspots’, subjects that until recently were known mainly from articles in technical journals with limited coverage of a much broader field, or from the Web sites of the institutions involved in the research and development. Only in the past few years has the subject received wider public notice with the publication of several overview books, as noted in Appendix B.
Book Structure The book is organized into three fairly independent parts, each approaching the subject from a different direction. There is some overlap, but you should find that each part is complementary. Therefore, linear cover-to-cover reading might not be the optimal approach. For some readers, the visions and critiques in Part III might be a better starting point than the abstract issues of Part I, or the technicalities in Part II. Part I sets the conceptual foundations for the later discussions. These first four chapters present mostly high-level overviews intended to be appropriate for almost anyone. The first chapter starts with the basics and defines what Web technology is all about. It also introduces the issues that led to the formulation of the Semantic Web initiative. Chapter 2 introduces the architectural models relevant to a discussion of Web technologies and defines important terminology. Chapter 3 discusses general issues around creating and managing the content and metadata structures that form the underpinnings of the Semantic Web. Finally, Chapter 4 looks at online collaboration processes, which constitute an important motivating application area for Semantic Web activities. Part II focuses on the technologies behind the Semantic Web initiative. These core chapters also explore representative implementations for chosen implementation areas, providing an in-depth mix of both well-known and lesser-known solutions that illustrate different ways of achieving Semantic Web functionality. The material is detailed enough for Computer Studies courses and as a guide for more technical users actually wanting to implement and augment parts of the Semantic Web. Chapter 5 provides layered analysis of the core protocol technologies that define Web functionality. The main focus is on the structures and metadata assertions used to describe and manage published data. Chapter 6 is an in-depth study of ontologies, the special structures used to represent term definitions and meaningful relationships. Chapter 7 introduces the main organizations active in defining specifications and protocols that are central to the Semantic Web. Chapter 8 examines application areas of the Semantic Web where prototype tools are already implemented and available.
xviii
Preface
Chapter 9 expands on the previous chapters by examining application areas where some aspect of the technology is deployed and usable today. Part III elevates the discussion into the misty realms of analysis and speculation. Chapter 10 provides an ‘insights’ section that considers the immediate future potential for Semantic Web solutions, and the implications for users. Chapter 11 explores some directions in which future Web functionality might develop in the longer term, such as ubiquitous connectivity and the grander theme of managing human knowledge management. Finally, the appendices supplement the main body of the book with a terminological glossary, references, and resources – providing additional detail that while valuable did not easily fit into the flow of the main text.
Navigation This book is undeniably filled with a plethora of facts and explanations, and it is written more in the style of a narrative rather than of reference-volume itemization. Despite the narrative ambitions, texts like this require multiple entry points and quick ways to locate specific details. As a complement to the detailed table of contents and index, each chapter’s ‘at a glance’ page provides a quick overview of the main topics covered in that chapter. Technical terms in bold are often provided with short explanations in the Appendix A glossary. Scattered throughout the text you will find the occasional numbered ‘Bit’ where some special insight or factoid is singled out and highlighted. Calling the element a ‘Bit’ seemed to convey about the right level of unpretentious emphasis – they are often just my two-bits worth of comment. Bits serve the additional purpose of providing visual content cues for the reader and are therefore given their own List of Bits in Appendix C. When referencing Web resources, I use the convention of omitting the ‘http://’ prefix because modern Web browsers accept addresses typed in without it. Although almost ubiquitous, the ‘www.’ prefix is not always required, and in cases where a cited Web address properly lacks it, I tried to catch instances where it might have been added incorrectly in the copyedit process.
The Author As an independent consultant in Sweden for 30 years, I have been responsible for software development, localization projects, and design-team training. A special interest was adapting an immersive teaching methodology developed for human languages to technical-training fields. Extensive experience in technical communication and teaching is coupled with a deep understanding of cross-platform software product design, user interfaces, and usability analysis. All provide a unique perspective to writing technical books and articles.
Preface
xix
Monthly featured contributions to a major Swedish computer magazine make up the bulk of my technical analyst writing at present. Coverage of, and occasional speaking engagements at, select technology conferences have allowed me to meet important developers. I also maintain several professional and recreational Internet Web sites, providing commercial Web hosting and Wiki services for others.
Collaborative Efforts A great many people helped make this book possible by contributing their enthusiasm, time, and effort – all in the spirit of the collaborative peer community that both the Web and book authoring encourage. Knowledgeable professionals and colleagues offered valuable time in several rounds of technical review to help make this book a better one and I express my profound gratitude for their efforts. I hope they enjoy the published version. My special thanks go to the many reviewers who participated in the development work. The Web’s own creator, and now director of the World Wide Web Consortium, Sir Tim Berners-Lee, also honoured me with personal feedback. Thanks are also due to the editors and production staff at John Wiley & Sons, Ltd. Personal thanks go to supportive family members for enduring long months of seemingly endless research and typing, reading, and editing – and for suffering the general mental absentness of the author grappling with obscure issues.
Errata and Omissions Any published book is neither ‘finished’ nor perfect, just hopefully the best that could be done within the constraints at hand. The hardest mistakes to catch are the things we think we know. Some unquestioned truths can simply be wrong, can have changed since we learned them, or may have more complex answers than we at first realized. Swedish has the perceptive word hemmablind, literally blind-at-home, which means that we tend not to see the creeping state of disarray in our immediate surroundings – think of how unnoticed dust ‘bunnies’ can collect in corners and how papers can stack up on all horizontal surfaces. The concept is equally applicable to not always noticing changes to our particular fields of knowledge until someone points them out. Omissions are generally due to the fact that an author must draw the line somewhere in terms of scope and detail. This problem gets worse in ambitious works such as this one that attempt to cover a large topic. I have tried in the text to indicate where this line is drawn and why. Alternatively, I might sometimes make overly simplified statements that someone, somewhere, will be able to point to and say ‘Not so!’. My excuse is that not everything can be fully verified, and sometimes the simple answer is good enough for the focus at hand. A book is also a snapshot. During the course of writing, things changed! Constantly! Rapidly! In the interval between final submission and the printed book, not to mention by the time you read this, they have likely changed even more. Not only does the existing software continue to evolve, or sometimes disappear altogether, but new implementations can suddenly appear from nowhere and change the entire landscape overnight.
xx
Preface
Throughout the development process, therefore, book material was under constant update and revision. A big headache involved online resource links; ‘link-rot’ is deplorable but inevitable. Web sites change, move, or disappear. Some resources mentioned in the text might therefore not be found and others not mentioned might be perceived as better. The bottom line in any computer-related field is that any attempt to make a definitive statement about such a rapidly moving target is doomed to failure. But we have to try.
Book Support and Contacting the Author The Internet has made up-to-date reader support a far easier task than it used to be, and the possibilities continue to amaze and stimulate me. Reader feedback is always appreciated. Your comments and factual corrections will be used to improve future editions of the book, and to update the support Web site. You may e-mail me at
[email protected], but to get past the junk filters, please use a meaningful subject line and clearly reference the book. You may also write to me c/o the publisher. Authors tend to get a lot of correspondence in connection with a published book. Please be patient if you write and do not get an immediate response – it might not be possible. I do try to at least acknowledge received reader mail within a reasonable time. However, I suggest first visiting the collaborative wiki farm (follow links from www.leuf.com/ TheSemanticWeb), where you can meet an entire community of readers, find updates and errata, and participate in discussions about the book. The main attraction of book-related Web resources is the contacts you can form with other readers. Collectively, the readers of such a site always have more answers and wisdom than any number of individual authors. Thank you for joining me in this journey. Bo Leuf Technology Analyst, Sweden (Gothenburg, Sweden, 2003–2005)
Part I Content Concepts
1 Enhancing the Web Although most of this book can be seen as an attempt to navigate through a landscape of potential and opportunity for a future World Wide Web, it is prudent, as in any navigational exercise, to start by determining one’s present location. To this end, the first chapter is a descriptive walkabout in the current technology of the Web – its concepts and protocols. It sets out first principles relevant to the following exploration, and it explains the terms encountered. In addition, a brief Web history is provided, embedded in the technical descriptions. Much more than we think, current and future technology is designed and implemented in ways that critically depend on the technology that came before. A successor technology is usually a reaction, a complement, or an extension to previous technology – rarely a simple plug-in replacement out of the blue. New technologies invariably carry a legacy, sometimes inheriting features and conceptual aspects that are less appropriate in the new setting. Technically savvy readers may recognize much material in this chapter, but I suspect many will still learn some surprising things about how the Web works. It is a measure of the success of Web technology that the average user does not need to know much of anything technical to surf the Web. Most of the technical detail is well-hidden behind the graphical user interfaces – it is essentially click-and-go. It is also a measure of success that fundamental enhancements (that is, to the basic Web protocol, not features that rely on proprietary plug-in components) have already been widely deployed in ways that are essentially transparent to the user, at least if the client software is regularly updated.
Chapter 1 at a Glance Chapter 1 is an overview chapter designed to give a background in broad strokes on Web technology in general, and on the main issues that lead to the formulation of the Semantic Web. A clear explanation of relevant terms and concepts prepare the reader for the more technical material in the rest of the book. There and Back Again sets the theme by suggesting that the chapter is a walkabout in the technology fields relevant to the later discussions that chapter by chapter revisit the main concepts, but in far greater detail.
The Semantic Web: Crafting Infrastructure for Agency Bo Leuf # 2006 John Wiley & Sons, Ltd
The Semantic Web
4
Resource Identifiers defines fundamental identity concepts, protocol basics, and how content can at all be located in the current Web by the user’s client software. Extending Web Functionality examines proposed ways to enhance the basic Web transport protocol, as well as protocol-independent methods. From Flat Hyperlink Model describes the current navigational model of the Web, especially the hyperlink, and highlights the areas where it is lacking. After a wishlist of Web functionality, To Richer Informational Structures explores strategies for extending the hyperlink model with background information about the content. The Collaboration Aspect explores one of the driving forces for a new Web, after which Extending the Content Model shows why a unified way to handle content is important in any extension. Mapping the Infosphere discusses ways that have been tried to map what is on the Web so that users can find what they are looking for. Well-Defined Semantic Models introduces why current lexical mappings are insufficient for the task, especially if searching and processing is to be automated.
There and Back Again The World Wide Web was conceived and designed as an open information space defined by the hyperlink mechanism that linked documents together. The technology enabled anyone to link to any other document from hyperlinks on a published Web page – a page anyone could see, and link to in turn. The whole system could self-grow and self-organize. No permissions were required to set up such links; people just linked to whatever other published resources they found interesting and useful. The only requirements to participate were a simple Internet connection and a place to put a Web page. This open nature is fundamental to many of the early design decisions and protocol implementations, sometimes in ways that were later obscured or marginalized. Bit 1.1
The Web is an open universe of network-accessible information
This definition of the Web, formulated by Tim Berners-Lee, provides in all its simplicity the most fundamental description of the Web’s potential. Open network access enables a potentially infinite resource, for people both to contribute to and use. The explosive growth of the Web and the content it mediates is in many ways a direct consequence of this design. It has given rise to a remarkable plethora of both content and functionality, sometimes unexpected. I am very happy at the incredible richness of material on the Web, and in the diversity of ways in which it is being used. There are many parts of the original dream which are not yet implemented. For example, very few people have an easy, intuitive tool for putting their thoughts into hypertext. And many of the reasons for, and meaning of, links on the web is lost. But these can and I think will change. Tim Berners-Lee (www.w3.org/People/Berners-Lee/FAQ.html), ‘inventor’ of the Web and director of the W3C.
Enhancing the Web
5
In addition, the original design had the goal that it should not only be useful for humanto-human communication but also support rich human–machine and machine–machine interactions. In other words, the intent was that machines would be able to participate fully and help in the access and manipulation of this information space – as automated agents, for example. Bit 1.2 The Web had the twin goals of interactive interoperability and creating an evolvable technology The core values in Web design are expressed in the principle of universality of access – irrespective of hardware or software platform, network infrastructure, language, culture, geographical location, or physical or mental impairment. Before going further into the nature of such interactions and the infrastructure that is to support them, we need to explore the fundamental issues of resource identity, and how naming schemes relate to the protocols used to access the resources.
Resource Identifiers The full interoperability and open-ended nature of the Web was intended to be independent of language, as evident in the way the design specified the universality of referencing resources by identity through the Universal Resource Identifier (or URI). The principle that absolutely anything ‘on the Web’ can be identified distinctly and uniquely by abstract pointers is central to the intended universality. It allows things written in one language to refer to things defined in another language. Properties of naming and addressing schemes are thus defined separately, associated through the dereferencing protocol, allowing many forms of identity, persistence and equivalence to refer to well-defined resources on the Web. When the URI architecture is defined and at least one dereferencing protocol implemented, the minimum requirement for an interoperable global hypertext system is just a common format for the content of a resource (or Web object). Anyone can create a URI to designate a particular Web resource (or anything, actually). This flexibility is at the same time both the system’s greatest strength and a potential problem. Any URI is just an identifier (a ‘name’ often opaque to humans), so simple inspection of it in isolation does not allow one to determine with certainty exactly what it means. In fact, two different URIs might refer to the same resource – something we often also run across in naming schemes in the ‘everyday’ world. The concept of unique identifiers finds expression in many fields, and is crucial to ‘finding’ and handling things in a useful way. Any identifier scheme assuredly defines a useful namespace, but not all schemes provide any useful dereferencing protocol. Some examples from the latter category are the MIME content identifier (cid ) or message identifier (mid ) spaces, the MD5 hash code with verifiable pure identity (often used as secure verification of file identity), and the pseudo-random Universally Unique Identifier (uuid ). They all identify but cannot locate.
6
The Semantic Web
The ability to utilize such namespace schemes in the URI context provides valuable functionality—as is partially illustrated by some peer-to-peer (p2p) networks, such as Freenet (described in the previous book, Peer to Peer). Even a simple persistent identity concept for connection-oriented technologies, for which no other addressable content exists, can prove more useful than might be expected. Witness the ubiquitous mailbox namespace defined by the ‘mailto:’ protocol – unfortunately named, however, since URIs are functionally nouns not verbs. The resulting URIs define connection endpoints in what is arguably the most well-known public URI space, persistent virtual locations for stores of e-mail messages.
Understanding HTTP Space The other most well-known public URI space is the HTTP namespace – commonly called ‘the Web’. It is characterized by a flexible notion of identity and supports a richness of information about resources and relating resources. HTTP was originally designed as a protocol for remote operations on objects, while making the exact physical location of these objects transparent. It has a dereferencing algorithm, currently defined by HTTP 1.1, but augmented by caching, proxying and mirroring schemes. Dereferencing may therefore in practice take place even without HTTP being invoked directly. The HTTP space consists of two parts: Domain Name, a hierarchically delegated component, for which the Domain Name System (DNS) is used. This component is a centrally registered top level domain (TLD): generic (gTLD, such as example.com) or national (ccTLD, example.se). Relative Locator, an opaque string whose significance is locally defined by the authority owning the domain name. This is often but need not (indeed should rarely) be a representation of a local directory tree path (relative to some arbitrary ‘root’ directory) and a file name (example: /some/location/resource). A given HTTP URI (or resource object identity) is commonly written as a URL, a single string representing both identity and a Web location. URL notation concatenates the parts and prefixes the protocol (as http:// ). As a rule, client software transparently maps any ‘illegal’ characters in the URL into protocol-acceptable representations, and may make reasonable assumptions to complete the abbreviated URL entry. In practice, the domain component is augmented by other locally defined (and optional) prefix components. Although still formally DNS arbitrated, the prefix is determined by the local authority. It is resolved by the DNS and Web (or other service) servers in concert. Most common is ‘www.’ but it may also be a server name, a so-called vanity domain, or any other local extension to the domain addressing scheme. A related important feature is to dereference the physically assigned IP number bound to a particular machine. Domain names therefore improve URI persistence, for example when resources might move to other machines, or access providers reallocate. However, persistence of locators in HTTP space is in practice not realized fully on the Web.
Enhancing the Web
Bit 1.3
7
URLs are less persistent overall than might reasonably be expected
Improving persistence involves issues of tool maturity, user education, and maturity of the Web community. At a minimum, administrators must be discouraged from restructuring (or ‘moving’) resources in ways that needlessly change Web addresses. The design of HTTP and the DNS makes addressing more human-readable and enables almost complete decentralization of resources. Governance is freely delegated to local authorities, or to endpoint server machines. Implementing a hierarchical rather than flat namespace for hosts thus minimizes the cost of name allocation and management. Only the domain-name component requires any form of formal centralization and hierarchical structure – or rather, only as currently implemented does it require centralized registrars and domain-name databases for each TLD. The domain name is, strictly speaking, optional in HTTP. It is possible, if not always convenient (especially with the trend to share IP in virtual hosting), to specify HTTP resource addresses using the physical IP number locally assigned by an access provider. Other protocol spaces, such as Freenet, in fact dispense with domains altogether and rely instead on unique key identifiers and node searches. As a feature common and mandatory to the entire HTTP Web, as it is currently used, the DNS root is a critical resource whose impartial and fair administration is essential for the world as a whole. Ownership and governance of the root of the DNS tree and gTLD subtree databases has in recent years been subject to considerable debate. The situation is only partially and nominally resolved under international arbitration by ICANN. The Semantics of Domain Names Another issue concerning gTLD allocation, with relevance to the concept of ‘meaning’ (or semantics) of the URI, is the design intent that the different domain categories say something about the owners. The original international gTLDs and their intended application were clear, for example: ‘.com’ for commercial organizations with a true international presence (those with presence just in a single national region were recommended to use instead countrycode domains). ‘.org’ for non-profit organizations with an international presence. ‘.net’ for international providers of network services, for example Web hosts or service providers. Such a division provides basic meta-information embedded in the URL in a way that is easy to see. However, so many U.S. companies registered under .com that the common perception became that .com designated a U.S. business. This skew was due mainly to the original strict rules concerning the allocation of .us domains by state, rather than allowing a company with a multi-state presence to use a single national domain. The .com domains became in fact the most popular on the Web, a status symbol no matter what the purpose. It also gave rise to the pervasive ‘dotcom’ moniker for any Internet-related
The Semantic Web
8
business venture. In a similar vein, though less so, the popular but mistaken association arose that .org and .net were U.S. domains as well. Administration by a U.S. central registrar only strengthened this false association. These three gTLDs are considered open domains, because there are no formal restrictions on who may register names within them. Anyone, anywhere, can therefore have a dotcom, dotorg, or dotnet domain – business, individual, whatever. Further confusion in the role of these gTLDs arose when domain names became traded commodities. Public perception tends to equate brand name with domain name, regardless of the TLD. Attractive names thus became a limited commodity and the effective namespace smaller. Most brand owners are likely to register all three at once. Bit 1.4
The TLD level of domain names currently lacks consistent application
The TLD system defines the root of the URLs that, as subsets of URIs, are supposed to provide an unambiguous and stable basis for resource mapping by resource owners. Reality is a bit more complex, also due to hidden agendas by the involved registrars. Therefore, over time, the original semantic division was substantially diluted, and essentially it is today lost from public awareness. However, the intent of at least one of the original seven gTLD was preserved: ‘.int’ is used only for registering organizations established by international treaties between governments, or for Internet infrastructure databases. Since the early Internet was mostly implemented in the U.S. and the administration of the gTLD names was then under U.S. governance, the other three original categories were quickly reserved for U.S. bodies: ‘.edu’ for universities and corresponding institutions of higher education that qualified, became in practice the domain for U.S.-based institutions. ‘.gov’ became reserved exclusively for the United States Government and its federal institutions. ‘.mil’ became reserved exclusively for the United States Military. Other countries must use qualified ccTLDs for the same purpose, such as .gov.uk. A further special TLD, ‘.arpa’ is provided for technical infrastructure purposes. In an apparent attempt to reclaim some inherent TLD meaning, new usage-restricted gTLDs were proposed in 2000. First implemented were .biz, .info, .pro, and .name, while .coop, .aero, and .museum are still pending (see FAQ at www.internic.net/faqs/ new-tlds.html ). Since 2004, ICANN has added .asia, .cat, jobs, .mail, .mobi, .post, .tel, .travel, and .xxx to the proposal list. Unfortunately, we already see semantic erosion in some of the newly implemented TLDs. Also, the addition of more TLD namespaces not only risks further confusion in public
Enhancing the Web
9
perception (.biz or .com?), but can be detrimental to the original concept of reducing the cost of namespace management. Brand owners may feel compelled redundantly to add further TLDs to protect against perceived misappropriation. Bit 1.5
Conflicting interests are fragmenting the Web at the TLD-level
Issues of ownership and intent involve complex and changing policy and politics, not easily pinned down in any lasting way, and often at odds with the Web’s underlying concept of universality. The resulting confusion, market pressures, and market conflict subtly distort the way we can usefully map the Web. In 2004, the W3C was, in fact, moved to state that the influx of new TLD subtrees was harmful to the Web infrastructure, or at the very least incurred considerable cost for little benefit. Detailed critique is given for some of these ‘special-interest’ proposals (see reasoning at www.w3.org/DesignIssues/TLD and www.w3.org/2004/03/28-tld ), for example: Implementing .mobi would seem to partition the HTTP information space into parts designed for access from mobile devices and parts not so designed. Such a scheme destroys the essential Web property of device independence. The domain name is perhaps the worst possible way of communicating information about the device. A reasonable requirement for adaptive handling of mobile devices is that it be transparent, by way of stylesheets and content negotiation. Content-filtering through TLD (as in .xxx) is unlikely to be effective, even assuming bestcase consensus, applicability, and binding international agreements on appropriate domain owners and suitable content in the respective TLD. We may also assume that many companies would merely redirect new special-interest domains (such as .travel ) to existing ones (an established .com, for instance). As it happens, even the superficially clear geographic relationship of the ccTLDs to the respective countries has been considerably diluted in recent years. The increased popularity of arbitrary businesses, organizations, or individuals registering attractive small-country domains as alternatives to the traditional dotcom ones causes more TLD confusion. In part, this practice reflects a preference for country codes that impart some ‘useful’ association, which can become popular in the intended contexts – for example, Tongan ‘.to’ as in ‘http://come.to’, or Samoan ‘.ws’ recast as meaning ‘website’ or ‘worldsite’. In part, it is a result of the increasing scarcity of desired or relevant dotcom names. This ‘outsourced’ registration, usually U.S.-based through some licensing agreement, generates significant foreign income (in U.S. dollars) for many small Pacific island nations, but assuredly it confuses the previously clear knowledge of ccTLD ownership. Pervasive Centralization Despite the decentralized properties inherent to the HTTP space design, the subsequent evolution of the Web went for a time in a different direction.
10
The Semantic Web
A needless fragmentation situation arose, with different protocols used to transfer essentially the same kind of text (or message) objects in different contexts – for example, e-mail, Web page content, newsgroup postings, chat and instant messaging. Such fragmentation, which leads to multiple client implementations, is confusing to the user. The confusion is only recently in part resolved by advanced multiprotocol clients. Web implementations also ignored to a great extent the interactive and interoperative design goals – much to the disappointment of the early visionaries who had hoped for something more open and flexible. This deficiency in the Web is part of what prompted the development of other clients in other protocols, to implement functionality that might otherwise have been a natural part of the HTTP Web. The pervasive paradigm of the Web instead became one of centralized content-providing sites designed to serve unilaterally a mass of content-consuming clients. Such sites constrain user interaction to just following the provided navigational links. Web pages thus became increasingly ‘designer’ imprinted, stylistic exercises ‘enhanced’ with attention-grabbing devices. Technology advances unfortunately tended to focus on this eyeball functionality alone. In fact, the main visitor metric, which tellingly was used to motivate Web advertising, became ‘page hits’ or ‘eyeball click-through counts’ rather than any meaningful interaction. Most Web-browser ‘improvements’ since the original Mosaic client (on which MS Internet Explorer, the long dominant browser, is based) have thus dealt more with presentational features than any real navigational or user-interaction aspects. Worse, much of this development tended towards proprietary rather than open standards, needlessly limiting the reach of information formatted using these features.
Revival of Core Concepts Despite the lackluster Web client implementations with respect to user functionality, the potential for interactive management of information on the Web remained an open possibility – and a realm in recent years extended by alternative peer architectures. The way the Internet as a whole functions means that nothing stops anyone from deploying arbitrary new functionality on top of the existing infrastructure. It happens all the time. In fact, Internet evolution is usually a matter of some new open technology being independently adopted by such a broad user base that it becomes a de facto new standard – it becomes ever more widely supported, attracting even more users. Bit 1.6 Internet design philosophy is founded in consensus and independent efforts contributing to a cohesive whole Principles such as simplicity and modularity, basic to good software engineering, are paralleled by decentralization and tolerance – the life and breath of Internet. Several open technologies have in recent years provided a form of revival in the field, with a focus on content collaboration. One such technology, explored in an earlier book, The Wiki Way, is a simple technology that has in only a few years transformed large parts of the visible Web and redefined user interaction with Web-published content.
Enhancing the Web
11
Wiki technology relies on the stock Web browser and server combination to make collaborative co-authoring a largely transparent process. It leverages the existing clientserver architecture into a more peer-based collaboration between users. Prominent examples of large-scale deployment are Wikipedia (www.wikipedia.com) and related Wikimedia projects, and many open-source support sites (sourceforge.net). Other extending solutions tend to be more complex, or be dependent on special extensions to the basic Web protocol and client-server software (such as plug-in components, or special clients). Yet they too have their validity in particular contexts. This tension between peer and hierarchical models was further explored and analyzed in considerable detail in the book Peer to Peer, along with the concept of agency. Although these peer technologies lie outside the immediate scope of the current book, some aspects drawn from these discussions are taken up in relevant contexts in later chapters.
Extending Web Functionality With the enormous growth of the Web and its content since its inception, it is clear that new implementations must build on existing content to gain widespread usage, or at least allow a relatively simple and transparent retrofit. This development can be more or less easy, depending on the intentions and at what level the change comes. So far, we have seen such changes mainly at the application level. Long sought is a change at a more profound level, such as a major extension to the underlying Web protocol, HTTP. Such a change could harmonize many functionality extensions back into a common and uniform framework on which future applications can build. Current HTTP does in fact combine the basic transport protocol with formats for limited varieties of metadata – information about the payload of information. However, because it is descended from the world of e-mail transport (and an old protocol), HTTP metadata support as currently implemented remains a crude architectural feature that should be replaced with something better. Bit 1.7 The Web needs a clearer distinction between basic HTTP functionality and the richer world of metadata functionality A more formalized extension of HTTP, more rigorous in its definitions, can provide such a needed distinction and thus bring the Semantic Web closer to reality. Extending HTTP HTTP was designed as part of a larger effort by a relatively small group of people within the IETF HTTP Working Group, but Henrik Frystyk Nielsen (the specification author) claims that this group did not actually control the protocol. HTTP was instead considered a ‘common good’ technology, openly available to all. In this vein of freedom, HTTP was extended locally as well as globally in ways that few could predict. Current extension efforts span an enormous range of applications, including distributed authoring, collaboration, printing, and remote procedure call mechanisms.
12
The Semantic Web
However, the lack of a standard framework for defining extensions and separating concerns means that protocol extensions have not been coordinated. Extensions are often applied in an ad hoc manner which promotes neither reusability nor interoperability. For example, in the variant HTTPS space, a protocol distinction is made needlessly visible in the URI. Although HTTPS merely implies the use of HTTP through an encrypted Secure Socket Layer (SSL) tunnel, users are forced to treat secure and insecure forms of the same document as completely separate Web objects. These instances should properly be seen as transparent negotiation cases in the process of dereferencing a single URI Therefore, the HTTP Extension Framework was devised as a simple yet powerful mechanism for extending HTTP. It describes which extensions are introduced, information about who the recipient is, and how the recipient should deal with them. The framework allows parameters to be added to method headers in a way that makes them visible to the HTTP protocol handler (unlike CGI parameters, for example, that must remain opaque until dealt with by the handler script on the target server). A specification of the HTTP Extension Framework is found in RFC 2774. (Among other sources, see userfriendly Web site www.freesoft.org/CIE/RFC/r to search and read RFC documents.) Otherwise, the most ubiquitous and transparent functionality change in the Web in recent years was an incremental step in the basic Web protocol from HTTP 1.0 to HTTP 1.1. Few users noticed this upgrade directly, but a number of added benefits quickly became mainstream as new versions of server and client software adapted. Examples of new features include virtual hosting (to conserve the IP-number space), form-based upload (for browser management of Web sites), and MIME extensions (for better multimedia support). Many would instead like to see something more like a step to ‘HTTP-NG’ (Next Generation) that could implement a whole new range of interoperable and interactive features within the base protocol. WebDAV is one such initiative, styled as completing the original intent of the Web as an open collaborative environment, and it is discussed in Chapter 4. Consider, after all, how little HTTP 1.x actually gives in the form of ‘exposed interfaces’, otherwise often termed ‘methods’ and known by their simple names. These methods are in effect the ‘action verbs’ (one of many possible such sets) applied to the ‘identifier nouns’ (URI) of the protocol. The main method is to access resources: GET is the core request for information in the Web. It takes the form of HTTP header specifying the desired information object as either the unique location (URL) or the process (CGI) to produce it. (A variant, HEAD, might support the return of header information but without data body.) GET is actually the only HTTP method that is required always to be supported. It has a special status in HTTP, in that it implements the necessary dereferencing operation on identifiers – it defines the namespace. As such, GET must never be used in contexts that have
Enhancing the Web
13
side-effects. Conversely, no other method should be used to perform only URI dereferencing, which would violate universality by defining a new namespace. Many ad hoc extensions and p2p-application protocols are based solely on GET. The GET-queried resource is expected to return the requested information, a redirection URI, or possibly a status or error message. It should never change state. In the request response, an appropriate representation of the URI-specified object is transferred to the client – not, as is commonly assumed, the stored literal data. Representations, encoding, and languages acceptable may be specified in the GET-header request fields, along with any specific client-side information. These and other factors affect both what is returned and the chosen format or encoding. Client-side input may be handled by two other methods: PUT is the method to store information at a particular Web location specified by a valid URL. It is structurally similar to GET, except that the header is also associated with a body containing the data to be stored. Data in the body comprise opaque bitstreams to HTTP agents. POST is in effect an indirect method to pass an information object to a Web server. It is entirely up to the server to determine both when and how to deal with it (process, store, defer, ignore, or whatever). The header URI specifies the receiving server agent process (such as a named script), possibly including a suggested URI for the object. Other methods sometimes seen are essentially only extensions of these basic three. It is optional for the responding HTTP agent (the Web server) to implement an appropriate process for any of these input methods. The following are some formally registered examples: CHECKOUT and CHECKIN are version-control variants corresponding to GET and PUT, but with the added functionality of locking and unlocking, respectively, the data object for access/change by other users. TEXTSEARCH and SPACEJUMP are extended forms of GET applied to search and map coordinate positioning. LINK and UNLINK are variants of POST to add and remove meta information (object header information) to an object, without touching the object’s content. In fact, it is relatively rare to see PUT used these days. Most servers are configured to deny such direct, external publishing requests, except perhaps in specific, well-authenticated contexts. Instead, POST is used to pass data from client to server in a more open-ended way, by requesting that a server-defined link be created to the passed object. Although, traditionally, POST is used to create, annotate, and extend server-stored information, it was successfully MTME-extended in v1.1 to serve as a generic data upload and download mechanism for modern browsers. The use of POST allows tighter server control of any received information. Perhaps more relevant to security, it gives server control over processing, storage location, and subsequent access. The client may suggest a storage URI, but the server is never obliged to use it. Even if a POST is accepted by the server, the intended effect may be delayed or overruled by subsequent processing, human moderation, or batch processing. In fact,
The Semantic Web
14
the creation of a valid link may not only be significantly delayed, it may never occur at all. Overall, the guiding design thought is that HTTP requests may be cached. From the perspective of the client/user, simple requests must be stateless, repeatable, and free of side-effects. Bit 1.8
HTTP is, like most basic Internet protocols, stateless
State tracking is, however, desired in many situations, but must then be implemented using message-based mechanisms external to the base protocol. Information stored clientside in Web-browser ‘cookies’ are but one example of such workaround measures. It must be noted that POST is strictly speaking neither repeatable nor free from sideeffect, and thus not a proper replacement for PUT (which is both). In fact, POST requests are by definition declared noncachable, even though it might prove possible in some situations to cache them with no side-effects. Modern browsers correctly warn the user if a POST request is about to be reissued. The reason is of course that a POST often alters information state at the server, for example to generate content change, or to commit a unique credit-card transaction. State considerations place serious constraints on possible functionality in the context of the current HTTP Web.
Extending in Other Layers Extending the transport protocol (HTTP) is by no means the only proposed solution to extending Web functionality. The growing adoption of XML, not just in preference to HTML as markup but crucially as a real language for defining other languages, provides an alternative extension framework. XML is more broadly application-managed than HTML, and its functionality definitions are independent of whether the underlying HTTP is extended or not. Implementations include message-based protocols to extend Web functionality by embedding the extension methods inside the message content passed by the base protocol. The use of XML for inter-company remote operations became prevalent in 2001, mainly because of its ability to encapsulate custom functionality and convey specialized content meaning. This kind of functionality extension is what led to the development of Web Services, a term implying standard functionality distributed across the Web. Even the markup change alone has direct potential benefits on existing Web functionality – for example, search. Most search engines read the format languages, usually HTML tags that may often be applied inappropriately from the point of view of logical structure. Consequently, the search results tend to reflect the formatting tags rather than actual page content as expressed in natural language. XML markup can express semantic meaning and thus greatly improve search-result relevancy. There are both pros and cons in the XML approach to extending the Web framework. Some of both aspects hinge on the fact that the implemented message passing often relies on
Enhancing the Web
15
the HTTP POST method. Embedding messages in GET headers imposes unacceptable message constraints. Extended headers, for example, might be passed as POST content and thus be opaque to the HTTP agent. When referring to a particular resource addresses, this embedded reference cannot be cached or stored in the way a GET-request URI can. A typical workaround is a syntax extraction procedure to let the agent recover a workable URI from the POST header response to a GET request. The solution allows HTTP agents not to consider content-embedded extensions when responding to arbitrary requests. The XML extension approach shares header opaqueness, seen from the point of view of HTTP agents (such as servers, proxies, and clients), with more proprietary extensions anchored in specific applications. Nonetheless, XML extension at least has the virtue of being an open standard that can be adopted by any implementation and be freely adapted according to context. It is ongoing work. Functionality extension is not simply about extending the protocol; other issues must also be considered. Content Considerations A major obstacle facing anyone who wishes to extend the usefulness of the Web information space to automated agents is that most information on the Web is designed explicitly for human consumption. Some people question what they see as a ‘geek’ focus on extending and automating the Web with new gee-wizardry. Instead, they suggest that most people seem happy with the Web as designed for human browsing, and do not necessarily have a problem with that. Most people, they say, are finding more or less what they want, and are finding other things of interest as they browse around looking for what they want. Just because current Web content is designed supposedly for humans does not mean that it is a ‘successful’ design, however. For example, published Web content commonly relies on any number of implied visual conventions and assumptions to convey intended structure and meaning. Many of these often complex stylistic elements are neither easily nor unambiguously interpreted by ‘most people’ even with explicit instruction, and much less is it possible to decode them by machines. Furthermore, such style conventions can vary significantly between different authors, sites, and contexts. Embedded metadata describing the intent behind arbitrary visual convention would not just aid machines, but could provide unambiguous cues to the human reader – perhaps as consistent client-side styling according to reader preference, or as cue pop-up. The advantage of such explicit guidance is most obvious in the case of browsers for the visually impaired, a group of human readers who often have significant problems dealing with existing content supposedly designed for them. Even in cases where the information is derived from a database, with reasonably well-defined meanings for at least some terms in its tabled layout, the structure of the presented data is not necessarily evident to a robot browsing the site. Almost any such
16
The Semantic Web
presentation has implicit assumptions about how to interpret and connect the data. Humans readers familiar with the material can deal with such assumptions, but unaided machines usually cannot. For that matter, the material might not even be possible to browse for a machine following embedded hyperlinks. Other, visual navigational conventions might have been implemented (such as image maps, form buttons, downloaded ActiveX controls, scripted mouse-state actions, flash-animated widgets, and so on). Bit 1.9 Machine complexities that depend on human interpretation can signal that the problem statement is incorrectly formulated A fundamental insight for any attempt to solve a technological problem is that tools and technologies should be used in appropriate ways. In cases where it becomes increasingly difficult to apply the technology, then it is likely that significant advances require fundamentally changing the nature of the problem. Instead of tackling the perhaps insurmountable artificial-intelligence problem of training machines to behave like people (‘greater AI’), the Semantic Web approach is to develop languages and protocols for expressing information in a form that can be processed by machines. This ‘lesser AI’ approach is transparent to the human users, yet brings information-space usability considerably closer to the original and inclusive vision. Another content issue is the proliferation of different content formats on the Web, which usually require explicit negotiation by the user. This barrier not only complicates and hides the essential nature of the information on the Web, but also makes the content more difficult to handle by software agents. Granted, the initial design of HTTP did include the capability of negotiating common formats between client and server. Its inclusion was based on the correct assumption that the ideal of a single content format would remain a vision obstructed by the wild proliferation of proprietary data formats. However, this negotiation feature has never really been used to advantage. The rapid adoption of HTML as the common formatting language of the Web did make the need less pressing, but the real reason is that the ever-larger list of possible formats made automatic negotiation impractical. Instead content format became classified (and clarified, compared to traditional extensionbased advertising) by importing the MIME-type concept from the realm of e-mail. Reading the MIME declaration tells the handler/client how to handle an otherwise opaque data stream, usually deferring interpretation to a specified client-side application. Although MIME declarations now formally refer to a central registry kept by IANA, there is no reason why the Web itself cannot be used as a distributed repository for new types. A transition plan would allow such a migration. Unqualified MIME types are in this scheme interpreted as relative URIs within a standard reference URI, in an online MIME registry.
Enhancing the Web
17
Services on the Web The issues of distributed services (DS) and remote procedure calls (RPC) constitute a kind of parallel development to the previously considered Web extensions, and are also considered a future part of the Semantic Web. At present, they coexist somewhat uneasily with the Web, since they operate fundamentally outside Web address space yet fulfill some of the same functionality as the proposed Web Services (WS). DS and RPC applications often use proprietary protocols, are highly platform dependent, and are tied to known endpoints. WS implementations function within the Web address space, using Web protocols and arbitrary endpoints. They also differ from DS and RPC remote operation work in that WS transactions are less frequent, slower, and occur between non-trusted parties. Issues such as ‘proof of delivery’ become important, and various message techniques can become part of the relevant WS protocols. Further discussion of these issues is deferred to later chapters. Instead the next few sections trace the conceptual developments that underpin the Semantic Web.
From Flat Hyperlink Model In the Beginning was the Hyperlink . . . The click-to-browse hyperlink, that core convention of text navigation, and ultimately of Web usability, is in reality a key-data pair interpreted in a particular way by the client software. It reads as a visible anchor associated with a hidden representation of a Web destination (usually the URL form of a URI). The anchor can be any structural element, but is commonly a selection of text or a small graphic. By convention, it is rendered visually as underlined or framed, at least by default, though many other rendering options are possible. Bit 1.10 Good user interface design imbues simple metaphors with the ability to hide complex processing The clicked text hyperlink in its basic underlined style might not have been visually very pleasing, yet it was easily implemented even in feature-poor environments. Therefore, it quickly became the ubiquitous symbol representing ‘more information here’. Functionally, the hyperlink is an embedded pointer that causes the client to request the implied information when it is activated by the appropriate user action. The pragmatics of this mechanism, and the reason it defined the whole experience of the Web, is that it makes Web addressing transparent to the user. A simple sequence of mouse clicks allows the user freely to browse content with little regard for its location. The concept is simple and the implementation ingenious. The consequences were farreaching. Figure 1.1 illustrates this concept. This ‘invention of the Web’ has generally been attributed to Tim Berners-Lee (knighted for his achievements in 2004, thus now Sir Tim), who also founded the World Wide Web Consortium (W3C, www.w3.org) in 1994. A brief retrospective can summarize the early development history.
18
The Semantic Web
Figure 1.1 Conceptual view of hyperlink functionality as it is currently implemented to form the World Wide Web. The interlinking hyperlinks in Web content provide a navigational framework for users browsing it
In the late 1980s, Tim led an effort with Robert Cailliau at the CERN nuclear research center in Switzerland to write the underlying protocols (including HTTP) for what later came to be known as the World Wide Web. The protocols and technologies were disseminated freely with no thought of licensing requirements. The early work on the Web was based on, among other things, earlier work carried out by Ted Nelson, another computer and network visionary who is generally acknowledged to have coined the term ‘hypertext’ in 1963 and used it in his 1965 book, Literary Machines. Hypertext linking subsequently turned up in several contexts, such as in online helpfiles and for CD-content navigation, but really only became a major technology with the growth of the public Web. The matter of public availability deserves further discussion. Although it may seem like a digression, the issues raised here do in fact have profound implications for the Web as it stands now, and even more so for any future Semantic Web. Patents on the Infrastructure Since hyperlink functionality is seen as a technology, like any other innovation, it is subject to potential issues of ownership and control. Even though the W3C group published Web protocols and the hyperlink-in-HTML concept as open and free technology (expressed as ‘for the greater good’), it was only a matter of time before somebody tried to get paid for the use of such technologies when the economic incentive became too tempting. The issue of patent licensing for the use of common media formats, client implementations, and even aspects of Internet/Web infrastructure is problematic. While the legal outcome of individual cases pursued so far can seem arbitrary, taken together they suggest a serious threat to the Web as we know it. Commercialization of basic functionality and infrastructure according to the models proposed by various technology stakeholders would be very restrictive and thus unfortunate for the usability of the Web. Yet such attempts to claim core technologies seem to crop up more often. In some cases, the patent claims may well be legitimate according to current interpretations – for example, proprietary compression or encryption algorithms. But in the context of the Web’s status as a global infrastructure, restrictive licensing claims are usually damaging.
Enhancing the Web
19
Further complications arise from the fact that patents are issued by respective countries, each with its own variant patent laws. Trying to apply diverging country-specific patents to a global network that casually disregards national borders – in fact, appears innately to transcend such artificial boundaries – seems impossible. It beomes especially difficult if one country’s patented technology is another’s publicdomain technology (public-key cryptography was but one early example of this quandary). In practice, U.S. patent and copyright interpretations tend to be enforced on the Web, yet this interim state of affairs is unacceptable in the long term because it suggests that the global Internet is owned by U.S. interests. If a particular technology is critical for a widely deployed functionality on the Web, allowing arbitrary commercial restrictions easily leads to unreasonable consequences. Many corporations, realizing this problem, are in fact open to free public licensing for such use (usually formally with the W3C). Sadly, not all business ventures with a self-perceived stake in Web technology are as amenable. Bit 1.11
Commercialization (thus restriction) of access cripples functionality
This view, while not popular among companies that wish to stake out claims to potential pay-by-use markets, seems valid for information-bearing networks. The natural tendency is for access and transaction costs to rapidly go towards zero. Unrestricted (free, or at least ‘microcost’) access benefits the network in that the increasing number of potential connections and relations exponentially increase its perceived value. Basic p2p network theory states that this value increases as a power relationship of the number of nodes – as 2n. Ownership issues should also be seen in relation to the general assault we can see on ‘free’ content. It is not easy to summarize the overall situation in this arena. Commercialization goals are pursued by some decidedly heavyweight interests, and often even long-established individual rights get trampled in the process. Legislation, interpretations, and application seem to shift almost daily. At least for hyperlink technology, the legal verdict appears to be a recognition of its importance for the common good. For now, therefore, it remains free. On the other hand, a failing perhaps of the current Web implementation is that it has no easy solutions to accommodate commercial interests in a user-friendly and unobtrusive way – the lack of a viable micropayment infrastructure comes to mind. Bit 1.12
Issues of ownership compensation on the Web remain unresolved
Without transparent and ubiquitous mechanisms for tracking and on-demand compensation of content and resource usage, it seems increasingly likely that the Web will end up an impoverished and fragmented backwater of residual information. Related to this issue is the lack of a clear demarcation of what exactly constitutes the ‘public good’ arena, one that despite commercial interests should remain free. Table 1.1
The Semantic Web
20
Table 1.1 The issue of ‘free’ usage and development or creation type Type
Open
Proprietary
Free to use freely Free to use, limited (licensed, non-commercial use) Pay to use (buy or license)
free stuff community rare, low cost
public good promotional usual case
outlines one way of looking at the dimensions of ‘free’ in this context. Not specifically located in it is the ‘free as in beer’ stance – that is, affordable for anyone. Hyperlink Usability What, then, are the characteristics of usability of the hyperlink? Of particular interest to later discussions on extensibility is the question: In what way is the usability lacking in terms of how we would like to use the Web? The current conceptual view of the hyperlink, as it has been implemented, has several implications for usability that are not always evident to the casual user. Current usage is deeply ingrained, and the functionality is usually taken for granted with little awareness of either original design intentions or potential enhancements. The major benefit is of hiding the details of URI and URL addressing from the user, behind a simple user action: clicking on a rendered hyperlink. Another benefit is the concept of ‘bookmarking’ Web resources for later reference, again through a simple user action in the client. These actions quickly become second nature to the user, allowing rapid navigation around the Web. On the other hand, despite the ingenious nature of the hyperlink construct, the deficiencies are numerous. They can be summarized as follows: Unidirectional linkage. Current implementations provide no direct means to ascertain whether hyperlinks elsewhere point to particular content. Even the simple quantitative metric that X sites refer to Web resource Y can be valuable as an approximate measure of authoritativeness. Mapping and presenting parent-child-sibling structures is also valuable. Backlink information of this nature can be provided indirectly by some search index services, such as Google, but this information is both frozen and often out-of-date. Unverified destination. There is no way to determine in advance whether a particular destination URL is valid – for example, if the server domain exists or the information is stored in the address space specified. The address can have been incorrectly entered into the hyperlink, or the server’s content structure reorganized (causing the dreaded ‘linkrot’ effect). Filtered access. Many sites disallow direct or ‘deep’ linking, either by denying such linked access outright or by redirecting to some generic portal page on the site. Content bookmarked in the course of sequential browsing may therefore not be accessible when following such links out of context. Other access controls might also be in effect, requiring login or allowed categories of users – and increasingly, the subscription model of content access.
Enhancing the Web
21
Unverified availability. Short of trying to access the content and receiving either content or error code, a user cannot determine beforehand if the indicated content is available – for example, the server may be offline, incapable of responding, or just applying filtered access. Relevance, reputation and trust issues. The Web is superbly egalitarian; anyone can publish anything, and all content hyperlinks have the same superficial value. In the absence of mechanisms for hyperlink feedback, the user has no direct way of determining the relevance or trustworthiness of the linked content. Visual spoofing of link destination is easy, which can lead users astray to sites they would never wish to visit, something often exploited by malicious interests on Web pages. As can be seen, the browsing user (also agents, and even the servers) could benefit from information about what exactly a hyperlink is pointing to, and from an indication of its status, operational as well as reputational. One-way linkage cannot provide this information. Existing tools are ‘out-of-band’ in the sense that they are not integrated into the browsing experience (or the client-server transactions), but instead rely on external, human-interpreted aids such as search engines or separate applications (say hyperlink checkers). Wouldn ’t it be nice if . . . Some features users might find useful or convenient when browsing the Web can be listed as a sampler of what the current infrastructure and applications either do not do, or do only with difficulty or limitations in special cases/applications. At this point, we are not concerned about the ‘how’ of implementation, only the user experience for some fairly simple and straightforward enhancements. Some of these features are hard to describe adequately in the absence of the ‘back-end’ data and automation, and they will depend greatly on commercial, political, and social decisions along the way as to what is desired or allowed. Context awareness, which would allow ‘intelligent’ client behavior. Ideally perhaps, semantic parsing of document context could reduce the need for user decision and simply present ‘most-probable’ options as distinctive link elements at the rendered location. Alternatively, the client might provide an on-the-fly generated sidebar of ‘related links’, including information about the site owner. Several proprietary variations of the theme are found in adware-related generation of ‘spurious’ hyperlinks in displayed content, which however is a more intrusive and annoying implementation in the absence of other cues. A simple example of a context feature for content is found in newer browser clients, where the user can highlight text and right-click to access Web search or other useful actions, as illustrated in Figure 1.2. Persistent and shareable annotations, where users can record private or public comments about Web content automatically attached to a given document (URI) and ideally internal location, in context. Public comments would then be shareable among an entire community of users – perhaps most usefully in goal-oriented groups.
22
The Semantic Web
Figure 1.2 A simple example of context awareness that allows a browser user to initiate a number of operations (such as Web search, Dictionary or Encyclopaedia lookup, or translation) based on a highlighted block of text and the associated context menu
Persistent and shareable ratings, which is a complement to annotations, providing some compiled rating based on individual user votes. A mouse-over of a link might then pop up a tooltip box showing, for example, that 67% of the voting users found the target document worthwhile. Of course, not everyone finds such ratings useful, yet such ratings (and their quality) might be improved by a consistent infrastructure to support them. More realtime data, which would result from the ability of Web clients and services to compile Web pages of data, or embed extended information, culled from a variety of sources according to the preferences of the user. Persistent and shareable categorizations, which would be a collaborative way to sort Web documents into searchable categories. The common theme in most wish lists and methods of addressing the deficiencies of the current Web model is that the issues are based on adding information about the content. Then the user client can access, process, and display (or use it) in ways transparent but useful to the user – ideally with some measure of user control.
To Richer Informational Structures The further information about the content that the previous shortcomings and wish list highlight include the metadata and relational aspects, some of which might be implemented as distributed services on the Web. One idealized extension of the originally unidirectional information flow could look as in Figure 1.3.
Enhancing the Web
23
Figure 1.3 Conceptual view of hyperlink metadata functionality as it might be implemented to form an enhanced Web. Metadata and relational data are gathered and presented to the user for each embedded link
The point in the enriched hyperlink model is that further information about each hyperlink destination is automatically gathered and processed in the background. This behavior is an extension of how some caching solutions already now parse the currently loaded content and pre-fetch all hyperlink-referenced content into a local cache, ready for the next user selection. In the new model, the client software would also be busy compiling information about the referenced content, ready to present this to the user even before a next destination is selected. Link metadata might be presented to the user as tooltip text, for example, or in ancillary windows on a mouse-over of the link. A very limited and static simulation of this feature can be seen in the use of the ‘title’ attribute in a hyperlink markup (as shown in Figure 1.4) – the difference is that this kind of metainformation indicator must be precoded by the content author as an attribute in the hyperlink tag itself.
The Collaboration Aspect The way that we have viewed the Web has generally been one of computer users browsing a web of content; independent readers of static information, or possibly also sole publishers of personal Web sites. This view was never the whole truth.
Figure 1.4 Static meta-information precoded by content author into the hyperlink tag of the content itself, using the ‘title’ attribute. When rendering the page, the browser pops up a ‘tooltip’ box to show the hidden text when the cursor is placed over the link
24
Bit 1.13
The Semantic Web
In the early days of the Web, considerable collaboration was the rule
What is interesting about the early collaborative effort is that it occurred in a smaller and more open community, with implicit trust relationships and shared values. Among those who maintained Web pages, compiling and publishing resource lists became a major part of the effort. To find information on the Web, a user would visit one or another of these lists of hyperlinks and use it as a launchpad, long before the idea of ‘portal sites’ became popular on the Web. The implicit metadata in such a reference represents the value assessment of the person creating the list. Such personal lists became more specialized and interlinked as their maintainers cross-referenced and recommended each other, but they also became vastly more difficult to maintain as the Web grew larger. Some directory lists, almost obsessively maintained, eventually grew into large indexing projects that attempted to span the entire Web – even then known to be an ultimately hopeless goal in the face of the Web’s relentless and exponential growth. One such set of lists, started as a student hobby by David Filo and Jerry Yang in 1994, quickly became a mainstay for many early Internet users, acquiring the name ‘Yet Another Hierarchical Officious Oracle’ (YAHOO). By 1996, Yahoo! Inc. (www.yahoo.com) was a viable IPO business, rapidly diversifying and attaining global and localized reach. Yahoo! quickly became the leader, in both concept and size, with a large staff to update and expand manually the categories from a mix of user submissions, dedicated browsing, and eventually search-engine database. Other search engines went the other way: first developing the engine, then adding directories to the Web interface. A more recent collaborative venture of this kind is the Open Directory project (www. dmoz. org). An offshoot to cross-linked personal lists and a complement to the ever-larger directories was the ‘WebRing’ (www.webring.org) concept. Sites with a common theme join an indexed list (a ‘ring’) maintained by a theme ringmaster. A master meta-list of categories tracks all member rings. To appreciate the size of this effort, WebRing statistics were in July 2002 given as 62,000 rings and 1.08 million active sites, easily capable of satisfying any user’s special interests indefinitely. However, the WebRing phenomenon appears to have peaked some years ago. presumably shrinking in the face of the ubiquitous search-engine query. The figures show a clear trend: 52,250 rings with 941,000 active sites in August 2003, and 46,100 rings with 527,000 sites in February 2005 The individual user of today tends increasingly to use the more impersonal and dynamic lists generated by search engines. An estimated 75% of user attempts to seek Web information first go through one of the major search engines. Although search-engine automation has reduced the relative importance of individual collaboration in maintaining the older forms of resource lists, it is still not the panacea that most users assume. Issues include incomplete indexing of individual
Enhancing the Web
25
documents, query-relevancy ranking, and handling of non-HTML and proprietary formats. Search engines also face the same problem as other approaches of scaling in the evergrowing Web. They are unable to index more than a small fraction of it. Estimates vary, but the cited percentage has been shrinking steadily – recently 10%, perhaps even less. The bottom line is that centralized indexing in the traditional sense (whether manual or automatic) is increasingly inadequate to map the distributed and exponential growth of the Web – growth in size, variety, updates, etc. Bit 1.14
New requirements and usage patterns change the nature of the Web
The adaptation of any network is driven by requirements, technology adoption, resource allocation, and usage – if only because new and upgraded resources usually target areas of greatest loading in order to maintain acceptable levels of service. Peer to Peer (a previous book) is replete with examples. Returning to collaboration, it has become more important to support new and more flexible forms – between individuals and groups, between people and software, and between different software agents. Loose parallels could be drawn between this adaptive situation for the Web and the changing real-world requirements on telephone networks: As first, subscriber conversation habits changed slowly over the years, and group conferencing became more common for business. Later, cellular usage became ubiquitous, requiring massive capacity to switch calls between line and mobile subscribers from many diverse providers. As dial-up Internet access became more common, individual subscriber lines and exchanges became increasingly tied up for longer intervals. Digitizing the network and applying packet switching to all traffic helped solve some congestion problems by letting separate connections coexist on the same circuit. In all cases, telecom operators had to adapt the network, along with associated technologies, to maintain an acceptable service in the face of shifting requirements. The Web is a virtual network. The underlying Internet infrastructure (and the physical access and transport connectivity under it) must constantly adapt to the overall usage patterns as expressed at each level. New collaboration technology has generally just deployed new application protocols on top of the existing Internet and Web ones, forming new virtual networks between the affected clients. Better collaboration support is achieved by instead enhancing the basic Web layer protocol. Other, proprietary solutions are possible (and some do exist), but making these features a fundamental part of the protocol ensures maximum interoperability. Just as we take for granted that any Web browser can display any standard-compliant Web page, so should clients be able to access collaboration functionality. Providing mechanisms for dealing with content meaning that will automatically provide intelligent mechanisms for promoting collaboration in all its forms is also one of the aims of the Semantic Web.
The Semantic Web
26
Figure 1.5 The traditional distinction between ‘‘document’’ and ‘‘message’’ is illustrated in the context of the client-server model, along with some terms relevant to requesting and viewing Web content
Extending the Content Model Discussions of ‘content’ without qualifications or explanations might be misleading for several reasons, and the term is easily misunderstood; this is why the terms ‘information’ and ‘information object’ crop up in these discussions. It is a common assumption that ‘Web content’ refers to the text or other media objects provided by Web pages and their related links – static information physically stored in welldefined server locations. In addition, historically, there has always been a distinction maintained between ‘messages’ on the one hand, and ‘documents’ (or files, or content) on the other (see Figure 1.5); that protocols are defined on top of messages in order to transport documents, and that further protocols are defined in terms of messages exchanged by previously defined protocols, and so on. Such recursive definitions, while sometimes confusing, occur all the time and can prove very useful. As implied earlier, p2p protocols that define separate virtual networks are defined by the exchange of underlying HTTP messages. However, especially in the context of XML, it is often better to view content as the sideeffect of an ongoing exchange of messages between various endpoints. It matters not whether these endpoints are servers of static Web pages, database servers, Web services, other users, or any form of software agent. Also, it does not matter whether the information is actually stored somewhere, compiled from several sources, or generated totally on the fly. Higher-level protocols are generally implemented as message-based transactions that can encapsulate information and pointers to information in many complex ways. Recall that requesting information on the Web, even in a basic protocol such as HTTP, only returns a representation of the information, encoded in some way acceptable at both endpoints – only a message, in effect, from the process providing it and thus inextricably bound to the protocol. Bit 1.15
Messages pass representations of information
The issue of information representation by proxy and subsequent interpretation is far more fundamental than it might seem. The concept has deep philosophical roots, and farreaching implications about how we know anything at all and communicate.
Enhancing the Web
27
This generalized view, that all content is the same exchange of messages expressed in a single protocol, amounts to a useful simplification. It can result in a merging of the current fragmentation of different protocols for e-mail, file transfer, newsgroup postings, instant messaging, chat, Web pages, and other ‘content’. The same client, based on such a harmonized protocol, could transparently handle arbitrary information objects in a consistent way from the user perspective.
Mapping the Infosphere Navigating the wealth of information on the Web is not a trivial task, even assuming that it is all openly accessible. Various methods have gradually increased or decreased in relative importance for users over the years. In roughly chronological order, these navigational methods are summarized in the following list: Resource lists present hyperlinks that particular users have collected and recommended on their own Web sites. An outgrowth of this resource was the ‘meta list’, or list of hyperlinks to sites that maintained such lists. The quality and comprehensiveness of such lists could vary immensely. Meta directories are large hyperlink lists compiled by dedicated staff into some kind of hierarchical category trees, often with the aim of being a comprehensive guide to all content. Examples are the Yahoo! Web Site Directory (www.yahoo.com), Google directory (directory.google.com) and the Open Directory Project (dmoz.org). A problem with hierarchical listings is classification, which can be arbitrary, and may or may not coincide with the user’s perception of an appropriate category. Search engines provide ad hoc lists of hyperlinks to Web pages (or other Internet resources) culled from indexed databases collected and updated by automated ‘spiders’ that systematically traverse the Web hyperlinks on each Web site they can reach. Search results are often ranked according to some opaque metric of ‘relevancy’, but depend critically on the user’s ability to formulate ‘optimal’ search criteria based on suitable terms, along with the quality of the indexing process. Special interest compilations lie somewhere between idiosyncratic lists compiled by individuals, and the comprehensive Web directories. They aggregate self-nominated, like-themed sites into interlinked Web rings. Each site in a ring can directly or indirectly refer to its neighbors through a ring index, each ring organized in a category hierarchy in some meta ring. Web services can automate and manage resource discovery and content retrieval (including compilation and recasting) on behalf of a user. Such services might incorporate ‘learning routines’ to adapt to each user’s history of requests to determine scope and relevancy criteria, for example. Currently, Web information is largely ‘atomistic’, with relatively few and arbitrary relations defined between the different information objects – see also the previous discussion about hyperlink usability. This deficiency and the constraints of the current hyperlink model clearly have a severe impact on user navigation. The lack also severely hampers the development of automated agents for such purposes.
28
The Semantic Web
Structure and Relationship Traditionally, maps of Internet resources in general, and the Web in particular, have by necessity been very simplistic and one-dimensional: Human-compiled hierarchy directories have the virtue of including some, albeit variable, amount of metadata based on human assessment of the link target and classification into categories. Expressed relational metainformation is an external construct, however, independent of any explicit hyperlinks between sites. Automated link-walk compilations result in maps that are in this case purely structural – a ‘snapshot’ of Web hyperlinks. While subsequent indexing of at least portions of content from each node does allow for term searches to create simple maps that reflect literal term usage (and thereby indicate term-related linking), the content semantics are largely ignored. We can illuminate this distinction and some of the issues involved by looking to some well-established conventions from paper publishing. Paper-published content is inherently linear, because it has its origins in oral narrative. Unlike transient speech, however, published sentences are frozen in a medium that the reader can to some extent perceive outside of time, freely choosing what to read at any time. This characteristic enables the concept of multiple entry points based on some overall mapping. Paper publishing thus developed structural and semantic aids to the reader to provide such entry points into the content: Chapters and sections. As a structural device, the division of a text into chapters and sections imparts a layer of semantics in that the logical division helps order content and implies relationships between the parts. Table of contents. Given an expressed logical structure, it is possible to provide a list of headings and corresponding numbered pages. The reader can use the table of contents to gain an overview of the content structure and quickly find specific content by following the logical divisions. Abstracts and summaries. Older novels commonly provided short abstracts for each chapter, and although this practice was later dropped in fiction, the abstract lives on in academic publications. Summaries (especially executive summaries) are a similar device. Both variants are commonly set apart from the main text by special styling (such as extrawide margins, vertical whitespace, and italic font). A listing of keywords may be considered as an extremely abbreviated summary. Index. From the other end (and it is not coincidence that the index is found at the back of a book), a list of terms and their page-location in the text constitutes an index. Creating a useful index is a highly developed art; it is not simply a compilation of occurrences of all the words. Instead It critically relies on the compiler’s understanding of relevancy and importance – of which words or phrases to include, which occurrences to index in terms of the semantic context, which occurrences to ignore, which relationships to highlight using categories and multi-level listings, just to name a few considerations. Other semantic-mapping contructs also exist in publishing, but those listed here suffice for the purpose of comparison with the Web. For example, comparing with the previous
Enhancing the Web
29
examples of Web maps, we can see a rough correspondence between the human-compiled category list and the table of contents. In passing, we may note that despite well-developed mechanisms for multiple entry points for the reader, not to mention advanced non-linear reading possibilities in electronic form, most written content maintains the fiction of linear narrative, from beginning to end. The search-engine list of links derived from automated indexing would perhaps approximate the index, except for the fact that such a list compilation is currently an index created solely by listing all occurrences of the search terms. It lacks the semantic order and weighting that the expert indexer imparts, and critically, the distinction between different usages and meanings – for example, between the noun ‘book’ and the verb ‘to book’ (as in make a reservation). Natural language is replete with ambiguities of usage that only information about semantic context can even begin to resolve. With clever programming, Web searches can at least be made tolerant of some common typographical mistakes and regional spelling variations, but these searches remain semantically ignorant. Proper logical markup of text structure in Web content can somewhat offset this analytical lack, but at the cost of reliance on external conventions that may or may not be correctly applied in particular instances. Text markup still does not convey inherent semantic information, only overlain syntactical. Although Google is commonly cited as ‘the best’ search engine, mainly due to the way it leverages backlinks to provide relevant results ranking, it is not hard to find consistently ‘bad’ or skewed results in simple searches. Such frustrations are often due to the engine’s strictly lexical basis combined with the skew effect of some kinds of linked content. For search words that also relate to product names, for example, Google ranks online shopping pages higher than any other results, probably because of the wealth of product links both on-site and from affiliate sites. Advanced syntax may therefore be needed to find information sites not devoted to selling products. Links, Relationship, and Trust The fact that object A unidirectionally links to object B says little about any real relationship between them. Any metainformation about this relationship is largely inferred by the user. For example, we are willing to believe in a high degree of relevance of a hyperlink on a page that refers to another page on the same site, even in the absence of any explicit declarations about it. Such trust may prove unfounded, however, especially if the link is ‘broken’ (that is, the URL is temporarily or permanently invalid). But the point is that the hyperlink itself contains no metadata whatsoever about the relationship or target status. At best, the contextual text around the link might explain something about the target page; at worst, it might be totally misleading. The situation is even less reliable when the link targets an object outside the site, because then the page/link author does not even have nominal control over the target object. Target site contact might have changed drastically since the link was created, for example, or the
30
The Semantic Web
target site might employ filtering or redirection that makes the direct reference invalid and the target hard or impossible to track down manually. Trust is also at issue when a link (URL) points to a Web page that defines an association between URL and URI – perhaps the easiest way to create and publish a URI Anyone can do it, and anyone can on the face of it misrepresent an association to a particular URI. Trust issues of this nature are probably best resolved by embedding currently accepted mechanisms based on content hashes and public key signing of the published assertion. The assertion can then be independently verified as unaltered and the publisher identity verified through some established chain of trust – in the worst case, just identified as the same publisher of other similarly-signed material.
Well-Defined Semantic Models When considering strategies to map information on the Web, one can begin by outlining some of the features that the mapped content, commonly expressed as ‘documents’ in these contexts, should exhibit or promote: It must be possible to define precisely a language (as a set of tokens, grammar, and semantics) in which to deal with ‘first class objects’ (which means document objects that are URI-referenced). Such a language is a prerequisite to processing. It must be possible to make documents in a mixture of languages (that is, allow language mixing). It is a fact of life that no single language can or will be used in all contexts. Extensibility is natural and alternate structures must be allowed to coexist. Every document should be self-defining by carrying the URI(s) of the language(s) in which it is written. This mechanism enables both humans and more importantly software to determine the appropriate methods for analyzing and processing the information. It must be possible to process a document while understanding only a subset of the languages (that is, partial understanding). In other words, full functionality must ‘degrade gracefully’ so that simpler processing can safely ignore the full set of features. The requirements in this list form the basis for XML development, for example, and it defines a first step towards enabling the encoding of semantic meta-information. The current Web recommendation is that XML should be used when a new language is defined. A new language (or, for that matter, new features extending an existing language) must then be defined as a new namespace within XML. Although this kind of language defining might be seen as akin to misguided attempts to (re)define all natural languages in terms of some grand metalanguage (as in the tradition of applying Latin grammar), as an artificial construct a priori applied to other artificial constructs, it can be both attainable and practical. Given the language, how do we deal with the meaning of what is expressed in that language? This issue is at the core of the Semantic Web, and a discussion around it forms the bulk of Chapter 2.
2 Defining the Semantic Web When discussing the Semantic Web, it is prudent to note that we are in fact dealing with but one of several interrelated emerging technology domains: Web Services (providing distributed functionality), Grid Services (integrating distributed resources), and the Semantic Web (mapping and managing the resources). This book’s primary focus is on the last of these. In order to map Web resources more precisely, computational agents require machinereadable descriptions of the content and capabilities of Web accessible resources. These descriptions must be in addition to the human-readable versions of that information, complementing but not supplanting. The requirement to be machine-readable is non-negotiable, because in future – in fact, practically already now – almost all software (and devices) will be innately Web-aware. Leveraging autonomic-computing technology enables agents and empowered devices to display independent activity and initiative, thus requiring less hands-on management by users. Already, an important expectation about software is that even ‘dumb’ applications can deal with basic Web connectivity in ways appropriate to their primary function.
Chapter 2 at a Glance This chapter introduces the architectural models and fundamental terms used in the discussions of the Semantic Web. From Model to Reality starts with a summary of the conceptual models for the Semantic Web, before delving further into detailed terms. The Semantic Web Concept section analyzes and motivates this particular extension to the Web. Representational Models reviews document models relevant to later discussions. The Road Map introduces some core generic concepts relevant to the Semantic Web and outlines their relationships. Identity must uniquely and persistently reference Web resources; interim solutions are already deployed.
The Semantic Web: Crafting Infrastructure for Agency Bo Leuf # 2006 John Wiley & Sons, Ltd
The Semantic Web
32
Markup must enable the distinction between the content representation and the metadata that defines how to interpret and process it. Relationships between data objects must be described. Reasoning strategies for inference engines must be defined. Agency on the Web can be implemented to apply reasoning on the other layers. Semantic P2P outlines the role of peer-to-peer technology in this context. The Visual Tour summarizes the high-level view. This section highlights The Map, The Architectural Goals and The Implementation Levels, and discusses the recent W3C architecture recommendations.
From Model to Reality The concept of a ‘semantic Web’ supposes a plan for achieving a set of connected applications for data on the Web in such a way as to form a consistently readable logical Web of data – the actual implementation of the Semantic Web (‘sweb’). The steps needed to get from the model of the Semantic Web to reality are many. With our current understanding of the model, the process will require many iterations of model refinement as we design the systems that will implement any form of Semantic Web. The Web was designed as an information space, with the goal not only that it should be useful for human–human communication, but also that machines would be able to participate and help. One of the fundamental obstacles to this vision has been the fact that most information on the Web is designed for human consumption. Even if derived from a database with well defined meanings (in at least some terms) for its columns, the structure of the data is rarely evident to a robot browsing it. Bit 2.1
When parsing data, humans read, while machines decode
Actually, humans do both: first decoding the syntactical representation, then applying built-in semantic rules and context inferences to interpret the meaning of the data. For machines to ‘read’ in this way, they also need semantic rules and inference logic. Initially leaving aside the ‘greater AI’ problem of training machines to behave like people (though assuredly some research along these lines is always being pursued), the ‘lesser AI’ and sweb immediate approach is instead to develop languages and codified relationships for expressing information in a machine-processable form. The machines must be able to do more than simply decode the data representations on the syntactical level; they must be able to perform logical processing and inference on well-defined meanings of the information tokens parsed from the data. Before going on to discuss possible models for this kind of machine enhancement, we might reasonably ask: What are the requirements on the content representation for meaningful reading to be feasible? The answer depends a fair bit on whom you ask. However, there is some consensus for the following content features: distinguishing markup from data; avoiding ambiguity; discouraging arbitrary variation.
Defining the Semantic Web
33
The linguist might at this point object that natural languages apparently thrive with a remarkably high degree of the latter two, ambiguity and variation – despite the many formal rigidities of syntax, grammar, and definition. Even the far more rigid and literal programming languages have their fair share of both characteristics. As it happens, people not only cope with ambiguity and variation, they actively crave and enjoy it, as evidenced in various forms of literature, poetry, and humour. It would seem that we as humans are predisposed forever to balance on the linguistic cusp between coherency and incoherency, playing constant games with the way we use language. This constant play and variation is one of the hallmarks of a natural, living language. On the other hand, computer languages (and by extension computer-logic implementations) have as a basic assumption the elimination of ambiguity, or at least they implement some formal abstraction so that the programmer need not consider it. Bit 2.2 Current computer logic design is blind to important parts of human communication It seems paradoxical that in this way we deliberately exclude the ability to automate the processing of a significant component of the information we produce. A few computer-languages do exist that are arguably exceptions to this literalistic constraint, inherently allowing some idiomatic variations and implicit referencing in ways that are similar to human languages. However, most programmers find such languages ‘difficult’ or ‘quirky’. While few might appreciate the ability to write poetry in code, such frivolous activities must have at least some bearing on the issue of creating programs that can deal with texts in ways that approximate human semantic processes. One practical field that demonstrates the necessity of semantic processing is that of machine translation of human languages. The limitations of syntactical and lexical parsing, not to mention the profound misconceptions of how humans process language meaning, have for decades confounded the practical realization of automated translation. At best, current implementations can achieve only tolerable results within narrow specialist fields, providing but a first approximation of meaning for a human reader.
The Semantic Web Concept Returning to the ‘plan’ to extend the Web (more of a vision, actually), we can start with a sort of ‘mission statement’ made by three of the top names behind the Semantic Web initiative: The Semantic Web is an extension of the current Web in which information is given well-defined meaning, better enabling computers and people to work in cooperation. Tim Berners-Lee, James Hendler, Ora Lassila
The Semantic Web
34
A reformulated version appears later to guide the formal W3C ‘initiative’: The goal of the Semantic Web initiative is as broad as that of the Web: to create a universal medium for the exchange of data. www.w3.org/2001/sw/Activity
To achieve this goal, existing Web content needs to be augmented with data (for example in the form of semantic markup) to facilitate its handling by computers, and with documents (or more generally, resources) that are intended solely for computers and software agents. It is envisioned that the resulting infrastructure, transforming the Web into the Semantic Web, will spur the development of automated Web services to fulfill a wide variety of purposes. Computers will be able discern the meaning of semantic data by following hyperlinks to term definitions and rule sets. Thus armed, they can reason about content logically and make decisions that are relevant to the context and to user expectations. Bit 2.3
Humans are the current Web’s semantic component
Today, humans are required to process the information culled from Web (Internet) resources to determine ultimately its meaning and relevance (if any) for the task at hand. Sweb technology would move some of that processing into software. Some writers want to (and often do) call such semantic processing simply AI. But after more than half a century of shifting expectations and broad popularization, the generic term ‘AI’ seems too loaded with old baggage and popular misconceptions to use without further qualification. For now, a better approach is to use more neutral and specific terms (such as ‘business logic’ or ‘semantics’) when discussing processing functionality in the context of the Semantic Web. Reasoning aside, we are not as yet dealing with any sort of ‘thinking’ software in the popularized ‘self-aware AI’ sense. Part III revisits the issue, however, and does discuss the broader AI aspects from a speculative perspective. The Near-term Goal More useful for conceptualization might be to see the envisioned change as moving from a Web that now reads like a random spread of open books, to a Web that instead reads like a well-structured database – not a centrally dictated order, though some might advocate such a vision, just a locally applied, machine-decipherable structure. In other words, the near-term goal is for a Web that is better suited to answering queries about the information published there. At its simplest level, such a change would mean that, for example, search engine results could be pre-filtered for some measure of ‘correctness’ in relation to the query’s semantic context. Such filtering is impossible in lexical searches. But perhaps more profoundly, it can mean the contextual structure to make actual user-invoked ‘searching’ less relevant. When asked what the ‘killer application’ of the Semantic Web would be, the clued-in proponents reply: The Semantic Web itself! They justify this reply by noting that the killer application of the current Web is precisely the Web itself – the Semantic Web is in this view
Defining the Semantic Web
35
just another expression of new innate functionality of the same magnitude. The capabilities of the whole are therefore too general to be thought about in terms of solving one particular problem or creating one essential application. The Semantic Web will have many undreamed-of uses and should not be characterized in terms of any single known one. Bit 2.4
The ‘Metadata Web’ must grow as the original Web did
As was the case with the growth of the original hyperlinked Web of content, it can be argued that until anyone can easily create metadata about any page and share that metadata with everyone, no true ‘semantic web’ will arise. The Semantic Web is most likely to develop first as expanding ‘islands’ of new functionality that over time become more generally adopted as various service providers realize the utility (and derived benefits or profits) of doing so. Such ‘semantic intrawebs’ are a reasonable first step to test the new approaches and tools, at a smaller yet fully functional scale, and therefore improve them before they are unleashed into the Web at large. To provide some context for this evolution, consider the following:
The original Internet was designed to pass messages between systems. The Web as we know it is largely about finding information. Web Services implement distributed functionality on the Web. The Grid intends to integrate functionality provided by the parts. The Semantic Web is concerned with describing the available resources, making the data accessible, and providing the agency to manage them.
Although very superficial descriptions, these items do capture something of the essence of the development. The latter three give rise to a qualitatively different dynamic nature in the Web than what we are currently used to.
Addressing some Practical Features Let us put the abstract vision into a few simple possible contexts by recalling some of the deficiencies and desired features discussed in Chapter 1 and seeing what a Semantic Web infrastructure could add in practice to the personal user experience. User-context tracking. Imagine that your Web client could (if you want it to) track your semantic context. For example, it might pre-filter search results to show only those hits with a high correlation to the meaning implied by previous contexts. Imagine further that meaningfully related search terms were automatically generated to catch resources that would otherwise be missed only because a particular search word did not occur or was spelled differently in the target. When browsing, the client might always highlight Web links that lead to resources with a high semantic correlation to the current context, based on background metadata processing. It might even suggest links not mentioned in the displayed content, and provide pop-up comments on demand.
36
The Semantic Web
Context-defined interactivity. Suppose the semantic context could bring to the foreground appropriate media types, or even different interactions with the data browsed. A table could be just a matrix of values, a graph could be just a bitmap picture of relations. Each could be a gateway to further information about each element/relationship. Elements could be manipulated in various ways to investigate other relationships. All would depend on the current or selected context. User-preference tracking. Anyone experiencing the ever-increasing prevalence of e-mail spam, and faced with the imponderable oddities of available junk-mail filters (such as in MS Outlook or in subscribed Web-based services), must wish for some simpler solution. Ideally one would simply tell a filtering agent to ‘ignore further mails of this kind’ and have it identify ‘similar’ content reliably, transparently adapting to a wide range of variation. Actually, what the user might often find more useful is the converse feature of the agent being able to highlight and give priority processing to desired mail that is semantically ‘valid’ and matches a current context or preference setting. The common thread here is an enhanced cooperation between resource server, browsing client, and user – where detailed content analysis and classification relevant to the current user is delegated to software agents, and where even the selection of data can depend on the context. Such enhancement can only work in a decent way if the software is capable of ‘seeing’ the meaning of content in a defined context, not just the lexical words and structural tags. Although this list may give the impression that we assume all such enhancements to be beneficial to and desired by the average user, this conclusion would be unfounded. At a minimum, as implied in the first item, the features should be optional and easily controlled by the user. At worst, otherwise useful features might be hijacked by content-publisher or advertising interests, reducing them to the equivalent of intrusive commercial spots in broadcast media. Bit 2.5
An important issue: Metadata functionality on whose terms?
The ideal vision presupposes that the user is in control and is the benefactor of the outlined enhancements. Although the distributed and peer architecture of the Web is inclined to promote (or at least allow) such a development, other strong interests in government and business often follow different, less benign agendas. Early precursors to context and tracking features have already appeared in limited ways, but they are not as yet semantically very clever. More critically, most of them are in fact not addressing user requirements at all, but are being pushed by business and advertising interests. Thus the user has little control over how these features work, or of when they are active. Often, the data collected is just reported back to the advertiser site, not helping the user in any way. The best that can be said for the context-tailored user profiles we have seen so far is that they may reduce the incidence of totally irrelevant banner ads – such as U.S. mortgage rates displayed to apartment dwellers in London. The user must of course first agree to such external profiling.
Defining the Semantic Web
37
What it Means for the Publisher In the Web of today, publishing content all too often means creating one-off presentations. Data are selected, a layout chosen, and the result stored as a page on a server somewhere. Updating changing data requires new presentations to be created and uploaded to the server. (Initially, I used the term ‘static’ here, but found it misleading on reading, as much such content is distressingly ‘dynamic’ to the user visually, in the form of animated images and intrusive shockwave-flash elements.) Moving to database-driven, dynamic pages (in the intended updated-content sense) is a step forward, in that it ensures presentations of the most current data. However, these pages remain locked to a particular layout. The publisher still needs to determine (or guess) the requirements of the visiting user, and design the database and page format accordingly. Trouble is, there is really no way of knowing what the visitor is looking for, or how the data will be used. Both issues are crucial, however, to a successful presentation design. Bit 2.6
Metadata allows ‘application-neutral’ publishing of data
The ideal for multi-purpose publishing is to provide ‘raw data’ that can be used and reused in a variety of contexts. Data descriptions can decouple the resource itself from issues of presentation, media type, and client behavior – user aspects otherwise to be determined by the collecting agent in the current context of the user. Granted, not everyone wants to publish raw data, and there are many legitimate reasons for retaining strict control of presentation, media type, and purpose. Clearly, the decision must remain with the publisher. However, by restricting access to fixed formats, the publisher also constrains the possible uses of the information – and incidentally risks that another, unsupported use will simply lift a snapshot of the data (soon out of date, perhaps) to embed in another presentation. Many useful applications of data are unforeseen (and unforeseeable) by the original publisher. The realistic choice for control thus lies somewhere between the two following extremes: Constrained access with complete control of presentation (‘link to this page’), yet paradoxically no control at all of ultimate information use and relevancy elsewhere. Unfettered access, with little direct control of how the information is presented, yet with the high and desired probability that users are linking to the resource directly to pick up the most current data on demand, instead of relying on stale static snapshots of earlier data. The latter approach does work in business, and it is gradually being adopted as companies realize the competitive benefits of allowing real-time access to the raw content data. For example, Amazon.com, with a long history of encouraged product linking from affiliate Web pages, has moved in the direction of more service-oriented support, such as for dynamic backlinks to Amazon servers. These define-once links automatically pull and embed up-to-date product images and pricing data when the affiliate page is viewed.
The Semantic Web
38
With similar access to manufacturer or vendor databases, resellers in general can quickly produce online catalogues as living documents, where product information and pricing is sourced and embedded in real time. Service providers can implement real-time inventory status, customer ordering, and support for sold services. All such solutions become easier to implement flexibly when consistent metadata exists alongside the required information. It is a question of business realizing that open-ended formats for published data do in fact benefit the bottom line. Extending in-house information to reuse by intermediaries greatly extends the reach of both the manufacturer and the vendor. Bit 2.7
People do not just want to read about it, they want to interact with it
Metadata descriptions and access to ‘raw’ data brings the virtual world closer to our real world, because we can then do more things online, on our own terms. As an example, imagine the added value of user/agent interaction affecting the presentation of a Web-published cookbook of recipes. The concept as such is fairly simple (and very popular), but a common reader issue is about scaling the size of the recipe to fit the dinner context. A sweb version that ‘understands’ content and context could, for example, recast browsed recipes to your current requirements (such as the specific number of people for dinner), offer substitutions for ingredients that you flag as unavailable (or point you to more suitable recipes), and suggest links to shops near your location that stock what the recipe requires. Smart agents could even warn that a particular recipe will not easily scale to the required size in the context of your kitchen, and the user interaction would start to resemble a conversation as you vary parameters and preconditions to find a better solution. Disentangling the Markup As noted, a core part of any Semantic Web implementation must concern the concept of machine-understandable documents. The ‘easy’ part is the clean separation of actual content (and critically, its meaning) from the presentation aspects – that is, from how the content is to be rendered by any given client in a particular context. Current HTML encoding practices routinely entangle structural and visual markup (see detailed discussion of markup in Chapter 5), despite the long availability of CSS to separate out the latter in a consistent and reusable way. Part of the blame for this state of affairs lies with still-inconsistent support of CSS in authoring and client implementations, and part lies with the original design of tag sets to aid computer typesetting. This legacy concern with typefaces, explicit visual layout, and the need to check the consistency of the document structure lives on in HTML. In addition, HTML is unable to address self-defining semantic markup; it can only deal with a limited and externally predefined syntactic structure. Bit 2.8
Useful semantic markup must be extensible and self-defining
We can never know (and therefore never standardize markup for) all possible contexts in advance. Usage evolves, and local yet portable extensions must be supported.
Defining the Semantic Web
39
Figure 2.1 Making Web content machine-understandable requires disentangling the meaningful content from the markup. Separating structure and presentation is only part of the answer. XML goes further by allowing self-defining tags to set apart meaningful parts of the text
Figure 2.1 depicts something of this relationship, adding a comparison with XML which, with its extensible and self-defining nature, appears to be the markup language of choice. Style sheets can be specified as CSS or XSL. Customized XML tags can set apart bits of text in ways that are semantically meaningful for understanding the context, and so provide a starting point for machine analysis of it. Even in XML, however, systematic reuse and repurposing of content requires that all presentational characteristics be stripped from the dataembedded markup. Although this task would seem straightforward, along the lines of CSS but applied to XSL, it in fact proves very difficult. One surprising result when attempting to formulate simple and consistent XSL rules for existing content is that much erstwhile ‘content XML’ idiom turns out to be mostly presentational in nature. Humans seem predisposed to mix presentation and semantics – it may be just the way we perceive the world. Perhaps our human perspective is so innately skewed to presentation scaffolding that we literally cannot isolate the semantic meaning without using strict tools to resolve it. Bit 2.9
Meaning is in the eye of the beholder
More research is needed into discovering how humans parse and classify meaning from various forms of presentation and communication. The later sections on ontologies, however, do provide some tools to grapple with this situation, and perhaps suggest a way out of the dilemma when we try to devise structures that can be processed by machines. Framework Constraint Any descriptive framework contains both visible and hidden assumptions, and XML is no different. For example, XML (as a derivative of SGML) presumes a tree structure when used to describe data. It expects, or tries to enforce, a universal linear representation of hierarchical structure, with important consequences for what it can and cannot describe easily in its syntax.
40
The Semantic Web
Figure 2.2 An illustration of the problem of excerpting content that is found in the context of linear markup. Missing or out-of-context tags break the syntactical sequence for correctly parsing markup
This artificial constraint can be a problem, especially when it comes to ‘excerpting’ content for use elsewhere, such as is easily envisioned when machines are to process information for remote presentation. In particular, the tree-based syntax makes assumptions about markup scope and span that can be undefined outside the linear context of the original text, as illustrated in Figure 2.2. Tree-based markup under any recognized SGML/XML convention is unable to easily represent all arbitrary structures. It fails to adequately represent overlapping or concurrent structures (as in the illustrated example). This fundamental limitation does not belittle the proven usefulness of the possible representations, but any system based on such a markup convention must work under constraint and therefore will not be able to handle easily other content structures. Another simple example is text citation, where a number of common conventions (such as annotations, quoter’s emphasis, ellipsis, and clarifying insertions) would perhaps translate to multiple layers of overlay markup, complicated by the additional fact that the cited content is no longer within the original markup context. Note that such constraints in practice also apply to much research and many constructs that relate to markup, as is shown in the following sections. The presumption of a tree for syntax (and hence for the structure of texts), however, is not necessary, despite its pervasiveness and seeming logic – it is only an assertion about data, not an inherent property of the data, and it is a partial description at best. Available processing technologies in the XML realm only obscure the issue. They rely on the same tree-based syntax and thus on the consequent presumption about the structure of data. Not all solutions are amenable to such descriptions. Embedding or Not Separating all the markup from the content, not just the presentational parts, has been proposed as a solution to several current markup issues. Relational pointers into the untagged data structure would then associate specified spans with the relevant metadata information. The additional overhead of maintaining parallel data structures and relational pointers between them is offset by the vastly greater ease of processing content in a more consistent and context-independent way. (A reasoned argument on this topic is ‘Embedded Markup Considered Harmful’ published by Ted Nelson in 1997 – see www.xml.com/pub/a/w3j/ s3.nelson.html.)
Defining the Semantic Web
41
Although the approach promises many advantages, it seems unlikely to be widely adopted any time soon. Interest in such a radical change of Web infrastructure has been low among content creators, so markup will continue to be embedded in the foreseeable future. We might see it as yet another example of how new technologies are constrained by previously chosen implementations, where identifiable advantages are offset by the perceived investment of globally adopting a revolutionary change. Parsing, Human versus Machine Even at the syntactical level, humans and machines parse data differently. In this context, we can speak of the ‘readability’ (or conversely, the ‘opacity’) of a document. Machine parsing inherently prefers binary material with no extraneous material or redundancy – a long opaque string of data that humans find hard or impossible to read. Such material is easy to process, using pattern-matching algorithms to extract tagged elements. Human readers prefer visual patterns, such as ample white space to group and separate semantic objects. They also prefer to have visual ‘noise’ (such as markup tags and other human-opaque linear elements) suppressed or transformed into typographical or layout conventions. Finally, humans experience a semantic dimension that enables, among other qualities, the appreciation of well-formed prose as an enhancement to easy reading. What we see in XML (and in fact the entire SGML family), when applied in isolation, is unfortunately a schizophrenic approach that concentrates on machine representation (that is, a syntactical tag system) to solve human-use issues on a case-by-case basis. In particular, the lack of a generic solution approach causes significant problems when trying to create syntax rules for machine processing. Informational Model Syntactic details aside, the Semantic Web approach must tackle the task of how information is to be represented. It is not just about improving the capability to search for information, although this goal is important enough given the pervasive use of search engines as a starting point for browsing. Just how important is search-engine technology? In only a decade the dominating method for information discovery and research in general has become user-initiated search on the Web – which is on the face of it an amazing development. It is often perceived as faster to look something up using a Web browser than to walk to the bookshelf and consult a printed reference work. It is assuredly convenient . . .when it works. Precise statistics are hard to come by (and interpret properly), but yearly surveys (such as reported at www.searchenginewatch.com) suggest that at least a tenth of all traffic to Web sites today is due to users first clicking on links from search-engine results – individual sites may report as many as three of four hits from such sources. Eight of ten first-time visits might arise from search results.
42
The Semantic Web
Users looking for products, for example, are far more likely to just plug the product name into a search engine (a surveyed third do so) and explore the resulting links, than to start by browsing shopping sites or to click on ads (less than a tenth together). On the other hand, many surveyed users follow a direct link to a site (such as a bookmark or typed-in Web address) and thus presumably already know where they are going. How did they learn of this address originally? In some cases, it is easy to surmise that casting a known brand name as a dotcom address might work; in other cases, it is not so evident. Although clearly significant, it remains highly conjectural how search-engine statistics actually correlate with the way people successfully find relevant sites to revisit (and hence generate the non-search related hits). Nevertheless, the assumption is driving a fast-growing, multi-million-dollar advertising industry devoted to favorable placement of ads on search result pages. Also, it motivates considerable efforts by webmasters to ‘optimize’ search engine ranking for their sites. Significant in this context is a 2001 survey that found that 97% of Fortune 100 companies had some type of site architecture problem that might give them problems being found by current search engines. The ‘might’ should be emphasized, however, since the most common problem related to missing or incorrect meta tags (that is, keywords to describe content) – these days the major search engines tend to ignore embedded metadata keywords in Web pages due to earlier misuse. Bit 2.10
Lexical (that is, syntactic) search characterizes the current Web
The pervasiveness and apparent usefulness of this model can easily blind us both to its inherent failings and to the potentials of alternative models. As the Web continues to expand, it becomes increasingly difficult for users to obtain relevant information efficiently. The common search strategies rarely scale well. The best results tend to come from engines that have useful ranking strategies such as backlink algorithms, or from engines that use a mix of sources, both human-classified directories and general indexing databases. All of these methods have known problems, however. The ultimate filter remains human inspection and judgement, and the effects of this extra user effort can be considerable. This last point is why result pages that provide a few lines of hit context are so useful. Yet even the best search-engines can be maddeningly inefficient in some situations and then generate mostly irrelevant results. Reasons for search engine inefficiency are detailed in Chapter 1. When search engines fail to satisfy, however, it is mainly because they are limited to reading the format tags and doing syntax analysis on a selection of published documents, sometimes not in their entirety. Despite indexing several billions of pages, such a database still represents only a small and arbitrary selection of total Web textual content – probably less than a tenth. The search situation for non-text content is far worse, though slowly improving. The greater volume of Web content thus remains inaccessible to automated remote indexing, for one or another reason. It matters little to the frustrated user whether a search is blocked due to restrictive access control, site-local archival solutions, non-standard navigation elements, or application-specific content formats (that lack Web markup).
Defining the Semantic Web
43
A new set of representational tools for metadata is required to make this immense repository of information accessible for a wide range of new applications. New interfaces are envisioned for users looking for information, and for applications that might process and compile the information for other, higher-level purposes. Requested information may not reside in any single Web location, or in a specific format, but semantic logic will enable programs to identify metadata and data elements on various sites, understand or infer relations, and piece them together reliably. Such will be the deployed infrastructure of the Semantic Web. Over time, much of the information published on the Web will tend to be packaged in metadata as a matter of course, and some resources may even lack any immediate ‘page’ location as such. Instead of searching for pages of information, the new model will involve framing queries to submit to information services, locally generating ‘content pages’ on the fly that satisfy the initial search requirements and provide further links to in-depth material. The result may be more like informal briefings or summaries. In fact, most retrieved information might be for agent consumption only and not reach the user directly at all. Bit 2.11
The implied query in searching will shift from ‘Where?’ to ‘What?’
This shift corresponds to a general shift in addressing from URL to URI. Strictly speaking, most users do not care where the information is stored, they just want useful answers and working functionality. The answer may be ‘out there’, but the user wants it ‘here’ and ready to use.
Representational Models In this section, we briefly review the main representational models that have relevance for the later discussion of the Semantic Web and related work. Of necessity, numerous technical terms are introduced here, albeit in very abbreviated contexts. Some of these topics are later discussed in detail in Chapters 5 and 6. Document Object Model The Document Object Model (DOM) is a tree-like model of objects contained in a document or Web page, defining its elements and allowing selective access to them for reading or manipulation. The abstraction of the DOM programming interface (API) provides a standardized, versatile view of a document’s contents that promotes interoperability. The DOM is an established tool for dynamically manipulating document data structures, not a data structure descriptor itself. The description is derived from the documentembedded markup. Like the SGML family of markup languages, the DOM also presupposes (or imposes) a tree-like structure in the data. However, a linear structure might not always be relevant or useful. Complementing the tree-based DOM API is an event-based Simple API for XML (SAX). It reports parsing events (such as the start and end of elements) directly to the application through callbacks, and does not usually build an internal tree at all. The application implements handlers to deal with the different events, much like handling events in a
The Semantic Web
44
graphical user interface. SAX is ideal for small-resource devices that need to locate specific elements in a single pass. As the custodian of DOM, the W3C maintains it as an open API to encourage standards compliance. See www.w3.org/D0M/Activity for an overview of its status. Early on, however, Microsoft added its own proprietary extensions to the DOM in implementations of its Internet Explorer browser. Since IE still remains the dominant desktop Web browser, this divergence creates interoperability problems for anyone creating Web content or programming applications that rely on DOM. Resource Description Framework The Resource Description Framework (RDF) is a general-purpose language for representing information in the Web. It is a common specification framework to express document metadata (that is, the meaning of terms and concepts used to annotate Web resources) in a form that computers can readily process. The purpose of RDF is to provide an encoding and interpretation mechanism so that resources can be described in a way that particular software can understand it. Bit 2.12
RDF is a core technology in the Semantic Web
In addition, most higher-level representations depend on or derive from the RDF model of expressing relationships and defining resources. RDF commonly uses XML for its syntax and URIs to specify entities, concepts, properties, and relations. The model is based upon the idea of making statements about resources in the form of a subject-predicate-object expression (called a triple). Essentially, the subject is the resource being described, the predicate is the resource’s trait or aspect that is being described, and the object is the value of that trait. This terminology is taken from logic and linguistics, where subject-predicate or subjectpredicate-object structures have very similar but clearly distinct meanings. Chapter 5 discusses RDF in greater detail. Machine-usable RDF statements tend to be very verbose by human standards due to the requirement that resource descriptions be unique. By contrast, human-readable statements to convey the same meaning are typically much shorter, but prove useless to a machine trying to parse the meaning. This difference derives from the way human readers intuitively understand and use context and implicit reference. Machine parsing needs all the relations explicitly spelled out. Schema The term schema denotes a map of concepts and their relationships. Invariably, such a map is both incomplete and simplified, often with a tight focus on a particular mapping need. The
Defining the Semantic Web
45
role of a schema as a representational model in the context of Web information is to mediate and adjudicate between human and machine semantics. Schemas based on RDF are described using RDFS, a language for defining RDF vocabularies, which specifies how to handle and label the elements. This schema view is intentionally broader than that of operational semantics. In the latter, the meaning of a token is what we want it to be for processing software to deal appropriately with the data. The former view is more likely to represents a structuralist stance in suggesting a sign relation between the signifier and the signified. As noted earlier, a common presumption is that the data structure is linear and tree-based. This constraint appears again in DTD and XML schema constructs. Nevertheless, XMLbased schemas are successfully used to model data in many database applications, and thus viewed as ‘superior’ for this purpose. XML provides the self-defining relationships; no need then to supply the data in half a dozen different formats, or to create and impose a new ‘standard’ exchange format. Regular tree grammar theory has established itself as the basis for structural specification in XML-based schemas. The other two constituents of a data model are the constraint specification, and the set of operators for accessing and manipulating the data. The field of constraint specification is still being studied, with different approaches such as ‘path-based constraint specification’ and ‘type-based constraint specification’. XML operators are available as part of XPath, XSLT, and XQuery. In Part II we examine much of the mechanics involved, mainly in Chapter 5. An alternative to constructing schemas using regular grammars is the rules-based approach. Its proponents often critique the otherwise uncritical adoption of grammarbased systems to define all schemas, sometimes retrofitting various ‘layer’ structures of pointers to handle the graph-like cases where the inherent tree-structure is inadequate. Some purists might beg to differ with the polarization often seen, especially when ‘rulesbased’ discussion segues into the claim of being a ‘semantic’ approach – to them, rule-sets are just another expression of a grammar, while semantics is about the means to choose between different rules and syntax. The pragmatic view for now is to follow current usage in this respect. However, the contrast between the two views can perhaps be illustrated by the following analogy, which is admittedly simplistic: Grammar-based schemas are similar to road maps; they try to locate precisely all mapped items on the grid. Implicit is the attempt to describe what is true for all of the data, all of the time. Getting from one point to another in such a map involves parsing the map and making decisions based on what one finds in the relationships (for instance, the topology and the available road connections). Rules-based schemas are like route descriptions; they try to express the things that make some particular information at some particular time and context different from other data of the same type. In practical terms, this operative approach is required for the data-entry and debugging phase. To summarize: the first tries to abstract away invariants, the second tries to find abstractions to express variations. In general, XML schemas and DTDs belong to the first category because they are concerned with mapping relationships.
The Semantic Web
46
Bit 2.13
Road-map and route-description schemas are largely complementary
This seeming dichotomy appears in most descriptive contexts, as classic state-oriented versus process-oriented views. Inspired design can leverage optimized use of each. The basic polarity between grammar-based and rules-based might usefully be expanded into a description of four layers to characterize schema languages, going from grammar to semantics. (If the example terms are not familiar at this time, please just ignore them for now.) Structural patterns. Processing is the binary result of matching the source against patterns expressed by a pattern language. Examples are DTD, TREX, RELAX, and the XSchema content model. Datatypes. Processing is the binary result of matching the source against datatypes expressed by a datatype language. Cited examples are XSchema datatypes and UML/XMI. Structural rules. Processing evaluates generalized expressions about structure against the source, and the results trigger actions – for example, to accept or reject the source as valid. Semantics. Processing is based on semantic rules that express meaning rather than formal structure. The rules may be constructed using RDFS, defining ontologies of meaning, such as for ‘business logic’ rules. A developer discussion relating to the last layer was once seen to make the implied criticism that our understanding of the requirements for a comprehensive semantic layer is at present still so rudimentary that a strict formalized description of it is not possible. Such a nihilistic view is contested by those who in fact work at developing semantic rules, but there is truth to the supposition that we still have much to learn.
The Road Map To explore the Semantic Web – design, implementation, and potential – we need to outline many aspects of its probable architecture, in particular what might be termed the essential components for it. In many respects, the outline describes an architectural plan untested by anything except thought experiments, more vision than substance, yet well on the way to becoming a reality. Even though many of the following concepts were introduced in Chapter 1, and are covered in far greater detail in later chapters, it is relevant here to summarize systematically the essentials as they pertain to the Semantic Web. Therefore, we adopt the viewpoint of the sweb developer looking for the general solutions that each concept might offer, rather than considering the deficiencies that are evident in current implementations. The following are then the ‘essential components’ of the Semantic Web.
Identity Handling resources on the Web requires a strong and persistent implementation of unique identity. In local contexts, looser or transient forms of identity might be acceptable, but such constructs will not work in the global Web context. The Uniform Resource Identifier is the generic solution, but it was not directly implemented in the early Web. The URI subset of Uniform Resource Locators (URL)
Defining the Semantic Web
47
was used to base global identity on an abstract ‘location’ instead – as protocol plus domain, plus local path or access method. However, the URL implementation has serious deficiencies in terms of persistency, among other things, in part due to its visible dependency on these arbitrarily and locally defined paths. No longer included in technical specifications, the now-deprecated term URL is only found in ‘informal’ usage and legacy documents. Bit 2.14
Actual location on the Web (URL) is less useful than it may seem
At best, a given URL represents a logical (and reasonably persistent) ‘location’ locally dereferenced by the resource server at the specified Internet domain. It may just reflect an actual but arbitrary (and likely to change) filesystem path for server storage. The user gains little and loses much by having the resource’s actual storage location or access method exposed in the URL. The unpredictable mobility or availability of Internet resources at specified URLs is an inconvenience at best, often resulting in cryptic error messages from the server instead of helpful redirection, but in the expanded context of the Semantic Web, explicitly exposing location becomes a critical flaw. For librarians (to take an example of a group on the front lines of implementing functionality in the direction of a semantic Web), it is a serious problem already. The deficiencies of the URL model compromise library services to patrons and impose unacceptably large burdens on catalog maintenance. Hence the desire for central repositories of uniquely named resource identifiers. The existence of a named identifier must be maintained even when the referenced resource no longer exists or is unavailable. Therefore, another URI subset, Uniform Resource Names (URN), has been under careful development by the IETF for some time, with the intent to provide persistent identities based on unique names within a defined namespace – as urn:namespace-identifier:named-resource. It is no coincidence that national libraries in many countries actively solicit the free registration of URN-URI addresses to online resources. A modification to the current DNS system, for example, would resolve current issues with changing resource URLs by providing dynamic translation from URN pointers to the actual server-relative locations (as expressed by updated URLs). Bit 2.15
Resource reference by name (URN) is better than by location (URL)
Most human-friendly locator schemes since the beginning of recorded history have been based on named references, leveraging the fundamental ability of humans to give and use names in a mapping or descriptive context. Importantly, names are portable. In practice, the URL and URN subsets do overlap somewhat. For example, a mailbox URL address for e-mail is also a URN, in that the domain-qualified target name is sufficient and
48
The Semantic Web
globally unique, and does not visibly depend on any server-local specifics such as path or method. The persistence requirement of URN schemes is more an outcome of the social structures that evolve to meet a common community need than a technological issue. Such evolution can be slow – in the case of librarians, too slow. Therefore, an interim solution was deployed: the Persistent Uniform Resource Locator (PURL) service. PURL can often be seen referenced in XML database-exchange constructions. The OCLC (Online Computer Library Center, Inc., www.oclc.org) maintains PURL Org (www.purl.org), a global resource in the form of a database of registered unique identifiers. OCLC is a non-profit, global library cooperative serving 41,000 libraries in 82 countries and territories around the world. The PURL service is in effect a URL redirector, as shown in Figure 2.3. A PURL URI associates a particular URL and XML term with a defined classification element, such as ‘http://purl.org/dc/elements/1.1/publisher’ (which, by the way, is a semantic definition). Resource references are defined using PURLs and references to the PURL server. The PURL service responds to the client request with the currently valid associated URL in its database, which the client can then use to access directly the resource. PURLs have been assigned to records cataloged in the Internet Cataloging Project, funded by the U.S. Department of Education to advance cataloging practices for Internet resources. This project represents the leading edge of MARC description of Internet resources, a mechanism by which computers exchange, use, and interpret bibliographic information. MARC data elements make up the foundation of most library catalogs used today. Bit 2.16
PURL is a pragmatic hack that implements URN-like functionality
Implemented as a transparent Web service, PURL is an example of a transitional infrastructure enhancement referenced using resource-embedded pointers. The expected syntax of URNs is clear enough to afford confidence that the syntax of transitional PURLs can be inexpensively and mechanically translated to the later URN form that is expected to be integral to the next-generation-internet infrastructure.
Figure 2.3 How PURL provides an intermediate step between the deficiency of impermanent URLs and the permanency of URNs. The PURL server acts as a redirector, dereferencing unique PURL identities into the currently valid URL
Defining the Semantic Web
49
In the meantime, the PURL model lends itself to distribution across the Net, with servers run by organizations with a commitment to maintaining persistent naming schemes (libraries, government organizations, publishers, and others).
Markup The syntactical component of the Semantic Web is the markup language that enables the distinction between the content representation and the metadata that defines how to interpret and process it. For various reasons, the current choice of markup language is XML, because among other things it fulfills the dual requirements of self-defining and extensible document descriptions. XML also provides well-trodden paths for normalizing markup and upgrading existing HTML content, possibly through transitional formats such as XHTML 1.0. Bit 2.17
XML is a fundamental building block for the Semantic Web
XML depends on URI, but in turn it is the foundation for most of the higher layers. The visible part of this component is the markup syntax, expressed as the reserved textpattern tags embedded in the document but invisible in human-use rendering. Primarily, markup language is for describing content structure. In practice, relevant parts of markup are client-interpreted and visually rendered in ways appropriate to accepted typographical and layout conventions. For non-visual rendering or other contexts, other forms of rendering would be more appropriate. When read aloud for the blind, ‘emphasis’, for example, would be a shift in inflection rather than an italic font style, and list or table items would be explicitly declared. XML is well developed, and many sources and books provide both overview and in-depth studies. The principles of XML markup are simple, as are the methods for extending the language with new markup – it is after all called ‘extensible’ for a reason. In the context of this book, it is sufficient to note the salient concepts: Reserved text-pattern ‘tags’ that define ‘containers’ in which content and metadata can be placed in a structured way. URI references to the resources that can provide information about how the XML tags used are to be interpreted. The basic syntax of XML is also found in other constructs, since it lends itself well to embedded definitions of more complex structures. An overview of relevant markup languages and protocols is found in Chapter 5.
Relationships Given identities and markup, a natural step is to attempt to identify and describe relationships between data published on the Web. Much useful and helpful functionality can be achieved by combining data from several independent sources.
50
The Semantic Web
Since most content is published independently in a variety of formats that cannot directly be parsed and ‘understood’ by agents, the sweb solution is to introduce a metadata framework in which such relationships can be expressed, and thus provide a query and conversion mechanism for content elements. Bit 2.18
Relationship expressions form the core of semantic query services
RDF and RDFS are languages well-suited to constructing relationship expressions. The simplest such relationship is the concept of equivalence, so that for example corresponding fields in different databases can be recognized as describing the same property, despite arbitrary field labels. Shared fields provide ways to answer queries that a single database cannot satisfy. Significant to the Semantic Web is the idea that such data relationships can be identified and published independently as metadata by anyone. Metadata resources enable agents to correlate data from multiple databases by identifying common fields and providing conversion rules. Actually, the application is even broader in that it enables the semantic wrapping of legacy resources and software in general. A simple representation of such a model is given in Figure 2.4. Related to this relationship layer, and implied by the concept of conversion, we also require logical rules somewhere. In the Semantic Web, such rules are assumed to be published documents themselves, linked to relevant data documents. Obviously, relationship rules can be added to the document stores of either database, but the openness of the concept frees us from a reliance on specific database owners to maintain such relationships. The relationship rules can be published as independent annotations, leveraging the ability of users to notice such relationships.
Figure 2.4 Relationships between data in independently published databases can be identified and published as a separate Web resource. Agents can thereafter use the published metadata to convert and correlate data from multiple, independent sources
Defining the Semantic Web
51
A logical layer enables a large number of important functions, for example:
Deduction of one type of document from a document of another type. Validation of a document against a set of rules of self-consistency. Authentication for allowed access and use of a document. Inference of valid relationships based on conversion and inference rules. Deduction of implicit knowledge from existing assertions. Conversion of a general query into more specific sub-queries. Resolution of a query by conversion from unknown terms into known ones.
Implementing such functionality is straightforward: a mechanism for quoting, simple predicate logic, and quantification expressions. It is the basis for creating a query language, and subsequently resolving the query into a result. RDF is the language framework chosen for this role in the Semantic Web.
Reasoning Application of logic requires some form of reasoning engine. Implementing high-level reasoning is in effect the holy grail of designing agents. This conceptual level is as yet underdeveloped in the Semantic Web – it is in crucial parts unspecified, unlike the lower protocol, language and logic layers. At this stage in sweb development, the architects are happy to defer the problem of reasoning. Currently, there is no definitive algorithm for answering questions – basically, finding proofs. Implementations rest on pragmatics. Construction of a proof is done in most applications according to some set of constrained rules; validation of a general proof is left to third parties. Thus the ‘proof’ of any query’s answer is given as the chain of assertions and reasoning rules invoked by transactions, with pointers to supporting material in other documents. RDF and derived knowledge structures have recently added well-defined model theories with standard associated proof theories. The design goal here is that practical queries be sufficiently constrained that the answers are computable – or are at least ensured to terminate in finite, reasonable time. The HTTP GET request will contain a proof that the client has a right to the response, while the response will be a proof that the response is what was asked for. A second goal is a form of adaptive forward compatibility. This interesting idea means that a reasoning engine of a given version should be able to reason its way through a descriptive document sufficiently well to be able to deduce how to read a document written in a later version of the language. A related idea is that of adaptive emulation, which means that an engine should be able to read a description of a second independent implementation (for similar functionality) well enough that it can process data produced by it.
Bit 2.19
Reasoning engines are a prerequisite for adaptive Web agents
Independent software agents with initiative need to reason about things and infer appropriate responses from context. They also need to adapt for forward compatibility.
The Semantic Web
52
In this context we become concerned with ontologies, at least in the sense that computer science has redefined the term for reasons of practicality, if only because we become concerned with vocabularies and some form of meaning. In addition, perhaps more significantly, we need a mechanism to ensure a reasonable expectation of repeatable results to the same query. The full scope of investigating ontologies is examined in Chapter 6.
Agency on the Web A diversity of algorithms exist today in search engines on the Web. Taken with the prooffinding algorithms in pre-Web logical systems, this abundance suggests that the Semantic Web will see many forms of agent evolve to provide answers to different forms of query and to perform a wide variety of tasks. In fact, it will probably be wrong to speak of ‘agent’ in the singular, isolated sense that we are used to when thinking of applications. Instead, user queries and delegated tasks seem likely to be handled by collaborative services using distributed instances of functionality all over the Web. A perceived ‘agent’ might therefore just be the local interface instance to transient swarms of collaborating services, each summoned for a particular task. This model is a multidimensional analog to the common Unix practice of ‘piping’ independent programs to process data sequentially. It may appear to be a complete application, but is only a transient combination. It seems probable that a wide range of agents will be tailored to particular contexts, to particular query scope and span constraints, and to various forms of cooperative ‘swarming’ to fulfill user requests. A veritable ecology of agent software, capable of varying degrees of reasoning and initiative, could in time become an ‘intelligence infrastructure’ (please excuse the hype) to overlay the virtual networks of data transfer that exist today. Bit 2.20
Adaptive Web agents enable interactive discovery of Web resources
First, as a rule, people are interested in the answers and results, not the detailed process of obtaining them. Second, people are interested in easy ways to assess and validate the results and the sources, but often it is enough to know an abstract metric of trust without necessarily and immediately seeing a detailed list of the actual sources and their credentials or reputations. In that context, people using the new Web would no longer be concerned about the location or format of information, or precisely which services need to be found and accessed, or for that matter if the requested information is even compiled in the required way. They would delegate the queries to their immediate agent interfaces and simply await ‘digest’ results. Alternatively, they might engage in a ‘conversation’ around the request, interactively refining precision while background processing keeps up to the changing criteria.
Semantic P2P Peer-to-peer technology is invoked implicitly when speaking of distributed collaboration and Web services. Simply providing computers on the Internet with more capabilities of processing and understanding the semantics of information will not be sufficient to bring Semantic Web technology to its full potential. Moreover, the way in which information is
Defining the Semantic Web
53
accessed on the Internet must undergo significant changes, away from the prevailing centralized publish-and-store-pages model. In part, the shift in infrastructure can be seen as a belated response to the shift in computer technology, with ever more powerful desktop PCs deployed and interconnected to provide an alternative to centralized mainframe storage and processing. We are seeing a general move towards a ‘ubiquitous computing’ infrastructure, and in consequence, ever distributed storage and retrieval methods – a complex tapestry of p2p topologies. Centralized storage and processing is unlikely to disappear completely – various contexts will either require or prefer such resources. However, even these traditional server architectures seem likely to connect increasingly to, and utilize, peer-based services to accomplish their tasks. After several pendulum swings between the extremes of centralized and distributed models, designs favoring hybrid solutions appear to be gaining acceptance in many application areas. Bit 2.21
Distributed p2p can make precise publishing location irrelevant
Just as resources will be accessed mainly by location-agnostic name (URN), publishing may often be to a named service rather than any specific server location. The key shift is that sweb resources, both data storage and processing, seem likely to be primarily distributed in adaptive, virtual networks. Who really cares where the data are stored and processor cycles are consumed, as long as it is all secure enough and available on demand? There are many advantages to encrypted and redundant storage across multiple machines on a global network, and such storage could easily become the norm for most cases. Prototype and commercial systems already exist on intranets and in the wild, and more are under development. As an example, the OceanStore Project (at oceanstore.cs.berkeley.edu), is a wellestablished distributed storage solution. It is intended to scale to billions of users around the world using an infrastructure of untrusted server resources. Technically, it is called solving distributed (and guaranteed) object location in a dynamic network. A related project is Tapestry, an overlay location and routing infrastructure that provides location-independent routing of messages directly to the closest copy of an object or service. It uses only point-topoint links, without centralized resources, and is self-organizing, fault-resilient, and loadbalancing. OceanStore looks to be a viable model for providing one aspect of infrastructure for the Semantic Web, that of location-agnostic ubiquitous storage and processing. The interesting part of the prototyping is that anyone can download working software and set up miniclusters on selected machines on the LAN or across the Internet. This process can form the islands that over time can merge into a complete global infrastructure, leveraging existing machine and network resources.
The Visual Tour Putting all the preceding into a coherent picture may not be the easiest task, for we often venture into terra incognito – uncharted regions about which we can only speculate, since neither the finished applications, nor sometimes even the data models they are to process, exist.
54
The Semantic Web
Figure 2.5 A ‘road map’ of sorts, based on a more detailed W3C diagram, which although dated, still reflects the relationships between different Semantic Web components examined in this book. See text for a discussion on how to read the arrows
Still, even the barest outline contours on the map serve a purpose by orienting the viewer in the relationships between prototypes, planned components, and vision. Not everyone is comfortable with simple graphs, but the subsequent discussions through this book might be augmented by the visual relationships they imply.
The Map Figure 2.5 illustrates the lay of the land, as it were, by describing some of the relationships between now-identifiable components in the vision of the Semantic Web. The diagram is based on a larger, more detailed road map prepared by the W3C some years ago, and describes the relationships as Why?/How? links. In other words, following any arrow answers ‘How do we realize the previous component?’ while going in the reverse direction answers ‘Why do we need the preceding component?’. Some of these components have been built (solid outline), at least as functional prototypes, others were planned (dashed outline). Still others remain preliminary or conjectural (no outline) until it is seen how such a system will work in practice. Some components, such as ‘trust’, require not just technological solutions, but also critical political decisions and social acceptance. Although this diagram may seem unusual due to the combined relationships, depending on which direction you read the arrows, and perhaps ‘backwards’ to some in its overall sense, it is still useful in seeing dependency relationships. You can read the ‘flow’ as in the following example, starting in an arbitrarily chosen node: To realize Agents (follow arrows emanating from this node), we need to implement Query Services, Trust mechanisms, and Logic systems. Why we need Agents (in this view, backtrack the arrow leading to this node) is to reach the goal of the Semantic Web. You may wish to refer back to this diagram from later discussions of specific components just to refresh their particular placement in the overall scheme of things. Variations and extensions of this diagram also appear later (in Chapter 5) to place other components in this same, goal-oriented scheme. Figure 2.6 shows an alternative ‘stacking’ view of layer dependencies, one of several variants popularized by Tim Berners-Lee (see www.w3.org/DesignIssues/diagrams/sw-stack-
Defining the Semantic Web
55
Figure 2.6 An alternative view of dependency relationships, as depicted in the ‘stack layer’ model popularized by Tim Berners-Lee
2002.png). It intends to show how certain implementation areas build on the results of others but, like most simple diagrams of this nature, it is only indicative of concept, not formally descriptive.
The Architectural Goals It is useful to summarize the goals introduced in the earlier text:
Identity, by which we understand URIs (not locator URLs). Data and Structuring, as represented by Unicode and XML. Metadata and Relationships, as defined in RDF. Vocabularies, as expressed in RDF Schema. Semantic Structure, as represented in Web Ontologies. Rules, Logics, Inferencing, and Proof, which to a great extent still remain to be designed and implemented to enable agency. Trust, as implemented through digital systems and webs of trust, and also requiring further development and acceptance. Now in the first years of the 21st century, we begin to see a clearer contour of what is to come, the implementation of components that recently were merely dashed outlines of conjectured functionality on the conceptual chart. Mindful of the rapid pace of sweb development in 2003 and 2004, the W3C issued a new recommendation document in late 2004, the first of several planned, that specifies more clearly the prerequisites and directions for continued development and deployment of swebrelated technologies: ‘Architecture of the World Wide Web’ (Vol 1, December 2004, w3.org/ TR/webarch/ ). The recommendation adopts the view that the Web builds on three fundamentals that Web Agents (which includes both human users and delegated software in the form of user agents) must deal with in a compliant way: Identification, which means URI addressing of all resources. Interaction, which means passing messages framed in standard syntax and semantics over standard protocols between different agents.
The Semantic Web
56
Table 2.1. W3C Principles and Good Practice recommendations for the Web Design Aspect
W3C Principle
W3C Good Practices
Constraints
Global Identifiers
Global naming leads to global network effects.
Identify with URIs. Avoid URI aliasing. Consistent URI usage. Reuse URI schemes. Make URIs opaque. Reuse representation formats. (New protocols created for the Web should transmit representations as octet streams typed by Internet media types.) Representation creators should be allowed to control metadata association. (Resource state must be preserved for URIs published as simple hypertext links.) A URI owner should provide consistent representations of the resource it identifies. URI persistence. A data format specification should provide for version information. An XML format specification should include information about change policies for XML namespaces. A specification should provide mechanisms that allow any party to create extensions. Also, specify agent behavior for unrecognized extensions. Separation of content, presentation, interaction. Link identification. Web-wide linking. Generic URIs. Hypertext links where expected. Namespace adoption. Namespace documents. QNames must be mapped to URIs. XML content should not be assigned Internet media type ‘text’, nor specify character encoding.
Assignment: URIs uniquely identify a single resource. (Agents: consistent reference.) (Transparency for agents that do not understand.)
Formats
Metadata
Interaction
Safe retrieval.
Representation
Reference does not imply dereference.
Versioning Namespace
Extensibility
Composition Hypertext data
XML-based data
XML Media type
Specifications Exceptions
Orthogonal specifications Error recovery (Consent may be by policy rules, based on informed not requiring interactive human user consent. interruption for correction.)
Agents must not ignore message metadata without the consent of the user. (Agents must not incur obligations by retrieving a representation.)
(Useful agent directives: ‘must ignore’ and ‘must understand’ unrecognized content.) (Usability issues.)
QNames are indistinguishable from URIs. (Reasons of correct agent interpretation.)
Defining the Semantic Web
57
Formats, which defines standard protocols used for representation retrieval or submittal of data and metadata, and which convey them between agents. The document goes on to highlight a number of Principles and Good Practices, often motivated by experiences gained from problems with previous standards. Table 2.1 summarizes these items. This table may seem terse and not fully populated, but it reflects the fact that W3C specifications are conservative by nature and attempt to regulate as little as possible. Items in parenthesis are expanded interpretations or comments included here for the purpose of this summary only. The recommendation document goes into descriptive detail on most of these issues, including examples explaining correct usage and some common incorrect ones. It motivates clearly why the principles and guidelines were formulated in a way that can benefit even Web content authors and publishers who would not normally read technical specifications. As with this book, the aim is to promote a better overall understanding of core functionality in order that technology implementers and content publishers achieve compliance with both current and future Web standards.
The Implementation Levels Two distinct implementation levels are discernible when examining proposed sweb technology, not necessarily evident from the previous concept maps: ‘Deep Semantic Web’ aims to implement intelligent agents capable of performing inference. It is a long-term goal, and presupposes forms of distributed AI that have not yet been solved. ‘Shallow Semantic Web’ does not aspire as high, instead maintaining focus on the practicalities of using sweb and KR techniques for searching and integrating available data. These more short-term goals are practical with existing and near-term technology. It is mainly in the latter category we see practical work and a certain amount of industry adoption. The following chapters examine the major functionality areas of the Semantic Web.
3 Web Information Management Part of the Semantic Web deals necessarily with strategies for information management on the Web. This management includes both creating appropriate structures of data-metadata and updating the resulting (usually distributed) databases when changes occur. Dr. Karl-Erik Sveiby, to whom the origin of the term ‘Knowledge Management’ is attributable, once likened knowledge databases to wellsprings of water. The visible surface in the metaphor represents the explicit knowledge, the constantly renewing pool beneath is the tacit. The real value of a wellspring lies in its dynamic renewal rate, not in its static reservoir capacity. It is not enough just to set up a database of knowledge to ensure its worth – you must also ensure that the database is continually updated with fresh information, and properly managed. This view of information management is applicable equally to the Web, perhaps the largest ‘database’ of knowledge yet constructed. One of the goals of the Semantic Web is to make the knowledge represented therein to be at least as accessible as a formal database, but more importantly, accessible in a meaningful way to software agents. This accessibility depends critically on the envisioned metadata infrastructure and the associated metadata processing capabilities. With accessibility also comes the issue of readability. The Semantic Web addresses this by promoting shared standards and interoperable mapping so that all manner of readers and applications can make sense of ‘the database’ on the Web. Finally, information management assumes bidirectional flows, blurring the line between server and client. We see instead an emerging importance of ‘negotiation between peers’ where although erstwhile servers may update clients, browsing clients may also update servers – perhaps with new links to moved resources.
Chapter 3 at a Glance This chapter deals with the general issues around a particular implementation area of the Semantic Web, that of creating and managing the content and metadata structures that form
The Semantic Web: Crafting Infrastructure for Agency Bo Leuf # 2006 John Wiley & Sons, Ltd
The Semantic Web
60
its underpinnings. The Personal View examines some of the ways that the Semantic Web might affect how the individual accesses and manages information on the Web. Creating and Using Content examines the way the Semantic Web affects the different functional aspects of creating and publishing on the Web. Authoring outlines the new requirements, for example, on editing tools. Publishing notes that the act of publishing will increasingly be an integral and indistinguishable part of the authoring process. Exporting Databases discusses the complement to authoring of making existing databases accessible online. Distribution considers the shift from clearly localized sources of published data to a distributed and cached model of availability based on what the data are rather than where they are. Searching and Sifting the Data looks at the strategies for finding data and deriving useful compilations and metadata profiles from them. Semantic Web Services examines the various approaches to implementing distributed services on the Web. Security and Trust Issues are directly involved when discussing management, relating to both access and trustworthiness. XML Security outlines the new XML-compliant infrastructure being developed to implement a common framework for Web security. Trust examines in detail the concept of trust and the different authentication models that ultimately define the identity to be trusted and the authority conferred. A Commercial or Free Web notes that although much on the Web is free, and should remain that way, commercial interests must not be neglected.
The Personal View Managing information on the Web will for many increasingly mean managing personal information on the Web, so it seems appropriate in this context to provide some practical examples of what the Semantic Web can do for the individual. Bit 3.1
Revolutionary change can start in the personal details
Early adoption of managing personal information using automation is one sneak preview of what the Semantic Web can mean to the individual. The traditional role of computing and personal information management is often associated with intense frustration, because the potential is evident even to the newcomer. From the professional’s viewpoint, Dan Connolly coined what became known as Connolly’s Bane (in ‘The XML Revolution’, October 1998, in Nature’s Web Matters, see www.nature. com/nature/webmatters/xml/xml.html): The bane of my existence is doing things I know the computer could do for me.
Web Information Management
61
Dan Connolly serves with the W3C on the Technical Architecture Group and the Web Ontology Working Group, and also on Semantic Web Development, so he is clearly not a newcomer – yet even he was often frustrated with the way human–computer interaction works. An expanded version of his lament was formulated in a 2002 talk: The bane of my existence is doing things I know the computer could do for me . . . and getting it wrong!
These commentaries reflect the goal of having computer applications communicate with each other, with minimal or no human intervention. XML was a first step to achieving it at a syntactic level, RDF a first step to achieving it at a semantic level. Dan’s contribution to a clearer perception of Semantic Web potential for personal information management (PIM) has been to demonstrate what the technology can do for him, personally. For example, Dan travels a lot. He received his proposed itineraries in traditional formats: paper or e-mail. Eventually, he just could not bear the thought of yet again manually copying and pasting each field from the itinerary into his PDA calendar. The process simply had to be automated, and the sweb approach to application integration promised a solution. The application-integration approach emphasizes data about real-world things like people, places, and events, rather than just abstract XML-based document structure. This sort of tangible information is precisely what can interest most users on the personal level. Real-world data are increasingly available as (or at least convertible to) XML structures. However, most XML schemas are too constrained syntactically, yet not constrained enough semantically, to accomplish the envisioned integration tasks. Many of the common integration tasks Dan wanted to automate were simple enough in principle, but available PIM tools could not perform them without extensive human intervention – and tedious manual entry. A list of typical tasks in this category follows. For each, Dan developed automated processes using the sweb approach, usually based on existing Web-published XML/RDF structures that can serve the necessary data. The published data are leveraged using envisioned Web-infrastructure technologies and open Web standards, intended to be as accessible as browsing a Web page. Plot an itinerary on a map. Airport latitude and longitude data are published in the Semantic Web, thanks to the DAML project (www.daml.org), and can therefore be accessed with a rules system. Other positioning databases are also available. The issue is mainly one of coding conversions of plain-text itinerary dumps from travel agencies. Applications such as Xplanet (xplanet.sourceforge.net) or OpenMap (openmap.bbn.com) can visualize the generated location and path datasets on a projected globe or map. Google Maps (maps.google.com) is a new (2005) Web solution, also able to serve up recent satellite images of the real-world locations. Import travel itineraries into a desktop PIM (based on iCalendar format). Given structured data, the requirements are a ruleset and iCalendar model expressed in RDF to handle conversion.
The Semantic Web
62
Import travel itineraries into a PDA calendar. Again, it is mainly a question of conversion based on an appropriate data model in RDF. Produce a brief summary of an itinerary suitable for distribution as plain text e-mail. In many cases, distributing (excerpts of ) an itinerary to interested parties is still best done in plain text format, at least until it can be assumed that the recipients also have swebcapable agents. Check proposed work travel itineraries against family constraints. This aspect involves formulating rules that check against both explicit constraints input by the user, and implicit ones based on events entered into the user’s PDA or desktop PIM. Also implicit in PDA/PIM handling is the requirement to coordinate and synchronize across several distributed instances (at home, at work and mobile) for each family member. Notify when the travel schedule might intersect or come close to the current location of a friend or colleague. This aspect involves extended coordination and interfacing with published calendar information for people on the user’s track-location list. An extension might be to propose automatically suitable meeting timeslots. Find conflicts between teleconferences and flights. This aspect is a special case of a generalized constraints analysis. Produce animated views of travel schedule or past trips. Information views from current and previous trips can be pulled from archive to process for display in various ways. A more technical discussion with example code is found at the Semantic Web Applications site (www.w3.org/2000/10/swap/pim/travel). Most people try to juggle the analysis parts of corresponding lists in their heads, with predictably fallible results. The benefits of even modest implementations of partial goals is considerable, and the effort dovetails nicely with other efforts (with more corporate focus) to harmonize methods of managing free-busy scheduling and automatic updating from Web-published events. Bit 3.2
Good tools let the users concentrate on what they do best
Many tasks that are tedious, time-consuming, or overly complex for a human remain too complex or open-ended for machines that cannot reason around the embedded meaning of the information being handled. The availability of this kind of automated tool naturally provides the potential for even more application tasks, formulated as ad hoc rules added to the system. For example: ‘If an event is in this schedule scraped from an HTML page (at this URL), but not in my calendar (at this URI), generate a file (in ics format) for PIM import’. The next step in automatic integration comes when the user does not need to formulate explicitly all the rules; that is, when the system (the agent) observes and learns from previous actions and can take the initiative to collect and generate proposed items for user review based on current interests and itineraries.
Web Information Management
63
For example, the agent could propose suitable restaurants or excursions at suitable locations and times during a trip, or even suggest itineraries based on events published on the Web that the user has not yet observed.
Creating and Using Content In the ‘original release’ version of the Web (‘Web 1.0’), a platform was provided that enabled a convenient means to author, self-publish, and share content online with the world. It empowered the end-user in a very direct way, though it would take a few iterations of the toolsets to make such authoring truly convenient to the casual user. In terms of ease-of-use, we have come a long way since the first version, not least in the system’s global reach, and practically everyone seems to be self-publishing content these days on static sites, forums, blogs, wikis, etc. (call it ‘Web 2.0’). But almost all of this Web page material is ‘lexical’ or ‘visual’ content – that is, machine-opaque text and graphics, for human consumption only. Creating Web content in the context of the Semantic Web (the next version, ‘Web 3.0’) demands more than simply putting up text (or other content) and ensuring that all the hyperlink references are valid. A whole new range of metadata and markup issues come to the fore, which we hope will be adequately dealt with by the new generation of tools that are developed. Bit 3.3
Content must be formally described in the Semantic Web
The degree of possible automation in the process of creating metadata is not yet known. Perhaps intelligent enough software can provide a ‘first draft’ of metadata that only needs to be tweaked, but some user input seems inevitable. Perhaps content authors just need to become more aware of metadata issues. Regardless of the capabilities of the available tools, authors must still inspect and specify metadata when creating or modifying content. However, current experience with metadata contexts suggests that users either forget to enter it (for example, in stand-alone tools), or are unwilling to deal with the level of detail the current tools require (in other words, not enough automation). The problem might be resolved by a combination of changes: better tools and user interfaces for metadata management, and perhaps a new generation of users who are more comfortable with metadata. A possible analogy in the tool environment might be the development of word-processing and paper publishing tools. Unlike the early beginnings when everything was written as plain text, and layout was an entirely different process relegated to the ranks of professional typesetters and specialized tools, we now have integrated authoring-publishing software that supports production of ready-to-print content. Such integration has many benefits to be sure. However, a significant downside is that the content author rarely has the knowledge (or even inclination) to deal adequately with this level of layout and typographical control. On the other hand, the issue of author-added metadata was perhaps of more concern in the early visions. Such content, while pervasive and highly visible on the Web, is only part of the current content published online.
The Semantic Web
64
Many existing applications have quietly become RDF-compliant, and together with online relational databases they might become the largest sources of sweb-compliant data – we are speaking of calendars, event streams, financial and geographical databases, news archives, and so on. Bit 3.4
Raw online data comes increasingly pre-packaged in RDF
The traditional, human-authored Web page as a source of online data (facts as opposed to expressed ‘opinion’) might even become marginalized. Aggregator and harvester tools increasingly rely on online databases and summaries that do not target human readers. Will we get a two-tier Web with diverging mainly-human-readable and mainly-machinereadable content? Possibly. If so, it is likely to be in a similar way to how we now have a two-tier Web in terms of markup: legacy-HTML and XML-compliant. Whether this becomes a problem or not depends – all things considered, we seem to be coping fairly well with the markup divide.
Authoring As with earlier pure-HTML authoring, only a few dedicated enthusiasts would even presume to author anything but the actual visible content without appropriate tool sets to handle the complexities of markup and metadata. Chapter 8 explores the current range of available tools, but it is clear that much development work remains to be done in the area. Minimum support for authoring tools would seem to include the following: XHTML/XML markup support in the basic rendering sense, and also the capability to import and transform existing and legacy document formats. Web annotation support, both privately stored and publicly shared. Metadata creation or extraction, and its management, at a level where the end-user can ‘easily’ do it. Integration of numerous types of information into a consistent view – or rather, into several optional views. Several of these criteria are addressed in at least the proof-of-concept stage by the W3C Amaya combined editor/browser client (see www.w3.org/Amaya/). The ideal Web 3.0 tool thus looks back at the original Web browser design, which was to have been both browsing and publishing tool. It adds, however, further functionality in a sensible way to provide users with the means to manage most of their online information management needs. In short, the user interface must be capable of visualizing a broad spectrum of now disparate local and online information items and views. Bit 3.5
The separation of browsing and authoring tools was unfortunate
In the Semantic Web, we may expect to see a reintegration of these two functions in generalized content-management tools, presenting consistent GUIs when relevant.
Web Information Management
65
The concept of browse or author, yet using the same tool or interface in different modes, should not be that strange. Accustomed as we are to a plethora of distinct dedicated ‘viewers’ and ‘editors’ for each format and media type, we do not see how awkward it really is. Tools can be both multimedia and multimode. This assertion should not be interpreted as sanctioning ‘monolithic bloatware, everything to everyone’ software packages – quite the opposite. In the Semantic Web, any application can be browser or publisher. The point is that the user should not have to deal explicitly with a multitude of different interfaces and separate applications to manage Web information. The functional components to accomplish viewing and editing might be different, but they should appear as seamless mode transitions. Working with Information One of the goals of the Semantic Web is that the user should be able to work with information rather than with programs. Such a consistent view becomes vital in the context of large volumes of complex and abstract metadata. In order to realize the full vision of user convenience and generation of simple views, several now-manual (usually expert-entry) processes must become automatic or semiautomatic: creating semantic annotations; linking Web pages to ontologies; creating, evolving, and interrelating ontologies. In addition, authoring in the Semantic Web context presumes and enables a far greater level of collaboration. Chapter 4 specifically explores collaboration space in this context.
Publishing Some people might debate whether creation and publishing are really very separate processes any more. Even in the paper-publishing world, it is more and more common for the content author also to have significant (or sometimes complete) control of the entire process up to the final ready-to-print files. The creation/publishing roles can easily merge in electronic media, for example, when the author generates the final PDF or Web format files on the distribution server. In the traditional content-creation view for electronic media, any perceived separation of the act of publishing is ultimately derived from the distinction between interim content products on the local file system, accessible only to a relative few, and the finished files on a public server, accessible to all. This distinction fades already in collaborative contexts over the existing Web. Now consider the situation where the hard boundary between user system and the Web vanishes – or at least becomes arbitrary from the perspective of where referenced information resides in a constantly connected, distributed-storage system. Interim, unfinished content may be just as public as any formally ‘published’ document, depending only on the status of applied access control.
The Semantic Web
66
Web logs are just one very common example of this blurring. Open collaboration systems, such as Wiki, also demonstrate the new content view, where there is no longer any particular process, such as a discrete FTP-upload, that defines and distinguishes ‘published’ content from draft copy. Bit 3.6
The base infrastructure of the Semantic Web supports versioning
With versioning support in place, managing interim versions of documents and collaboration processes online becomes fairly straightforward.
Exporting Databases Increasingly important, and a complement to explicit authoring, are technologies for exporting existing database content to the Semantic Web. In fact, a significant frustration in the business and consumer world, and consequent disenchantment with Web technology in general, has been the lack of direct access to business databases. Customers must often manually browse to another company’s site, drill down to the sought-for published document, and in worst case, parse a downloaded and hard-to-read PDF or MS Word document for the required information. A frozen PDF rendition of a database selection of records might in addition be critically out of date. Bit 3.7
Online business requires online database access
It is that simple, and any other offline solution, including Web-published static extracts (HTML tables or PDF) simply will not do. That more companies on the Web do not understand this issue can only be attributed to the ‘printed brochure’ mentality. Data export, or rather its online access, is easiest if the database already is RDF compliant, because then the problem reduces to providing a suitable online interface for the chosen protocol. Then customers can access realtime data straight from the company database.
Distribution In the traditional Web, it would seem that content is published to one specific location: a Web server on a specified machine. This content-storage model, however, is a serious oversimplification. Distribution was an issue even in the early Internet, and for example USENET is based on the idea of massive replication across many connected servers, motivated by the fact that overall bandwidth is saved when users connect to their nearest news server instead of to a single source. For reasons of reliable redundancy and lessened server loading, mirror sites also became common for file repositories and certain kinds of Web content. Such a visible mirror-server model requires a user to choose explicitly between different URLs to reach the same resource. However, transparent solutions for distributed storage and request fulfilment that reference a single URL are much more common than the surfing user might imagine.
Web Information Management
67
Content Caching or Edge Serving In particular, the growth of very high-demand content, such as provided by search engines, large corporations and institutions, and by popular targets such as major news sites, Amazon.com and eBay, all rely on transparent, massively distributed content caching. Behind the scenes, requests from different users for content from the same stem URL are actually fulfilled by distributed server clusters ‘nearest’ their respective locations – an approach often termed ‘edge’ serving. An entire infrastructure fielded by specialized companies grew up to serve the market, such as Akamai (www.akamai.com) that early on made a place for itself on the market by deploying large-scale solutions for aggressively distributed and adaptive storage. Distribution is also about functionality, not just serving content, and about having clusters or swarms cooperatively implement it. In the Semantic Web, we might be hard pressed to separate cleanly what is ‘content’ and what is ‘functionality’. The rendered result of a user request can be a seamless composite constructed by agents from many different kinds of Web resources. Storage can be massively distributed in fragmented and encrypted form as well, so that it is impossible to locate a particular document to any individual physical server. In such a case, user publishing occurs transparently through the client to the network as a whole instead of a given server – or rather to a distributed publishing service that allocates distributed network storage for each document. Retrieval is indirect through locator services, using unique keys generated from the content. Retrieval can then be very efficient in terms of bandwidth usage due to parallel serving of fragments by swarms of redundant hosts.
Syndication Solutions An area related to publishing and distribution is known as syndication, which in one sense, and as commonly implemented, can be seen as a user-peer form of mirroring. Syndication of content, however, differs from mirroring of content in the way the name (borrowed from news media and broadcast television) implies; it distributes only subscriber-selected content from a site, not entire subwebs or repositories. Syndication is currently popular in peer-to-peer journalism (p2pj) and for tracking new content in subscribed Web logs (commonly called ‘blogs’). Web syndication technology is currently based on two parallel standards, both with the same RSS acronym due to historical reasons of fragmented and misconstrued implementations of the original concept: Really Simple Syndication (RSS 0.9x, recently re-engineered as RSS 2.x to provide namespace-based extensions) is based on XML 1.0. It is now used primarily by news sites and Web logs to distribute headline summaries and unabridged raw content to subscribers. This standard and its development are described at Userland (see backend.userland.com/ rssO91 and blogs.law.harvard.edu/tech/rss). RDF Site Summary (known as RSS 1.x) is a module-extensible standard based on RDF, in part intended to recapture the original semantic summary aspects. It includes an RDF Schema, which is interesting for Semantic Web applications. The home of this standard is found at Resource.org (see web.resource.org/rss/1.0/for details). Official modules include Dublin Core, Syndication and Content. Many more are proposed.
68
The Semantic Web
Somewhat conflicting perspectives exist on RSS history (see rsswhys.weblogger.com/ RSSHistory). In any case, we find in all about nine variants of RSS in the field, adhering in greater or lesser degree to one of these two development forks. The name issue is especially confusing because RSS 0.9 was originally called RDF Site Summary when it was deployed in 1999 by Netscape (the company). It then provided the capability for ‘snapshot’ site summaries on the Netscape portal site for members wishing to advertise their Web site. Although conceived with RDF metadata summary in mind, the actual Netscape implementation dropped this aspect. Other developers and RSS portal sites later spread the concept, leveraging the fact that RSS could be used as an XML-based lightweight syndication format for headlines. Some advocates redefine RSS as ‘just a name’ to avoid the contentious acronym interpretation issue altogether. Others note the important distinction that the two parallel standards represent what they would like to call ‘RSS-Classic’ and ‘RSS 1.0 development’ paths, respectively: MCF (Meta Content Framework) ! CDF (Channel Definition Format) ! scriptingNews ! RSS 0.91 ! RSS-Classic. MCF ! XML-MCF ! RSS 0.90 ! RSS 1.0. Be that as it may. The formats can to some extent be converted into each other. In any case, a plethora of terms and new applications sprang from the deployment of RSS: Headline syndication, as mentioned, carries an array of different content types in channels or feeds: news headlines, discussion forums, software announcements, and various bits of proprietary data. Registries provide classified and sorted listings of different RSS channels to make user selection easier. Aggregators decouple individual items from their sites and can present one-source archives of older versions of content from the same site, and in some cases with Scrapers, advanced searching and filtering options. Scrapers are services that collect, ‘clean’, and repackage content from various sources into their own RSS channels. Synthesizers are similar to Scrapers, except that the source channels are just remixed into new ones, perhaps as a user-configured selection. Somewhere along the way, the ‘summary’ aspect was largely lost. Even as RSS, for a while, became known as Rich Site Summary (v0.91), it was morphing into full-content syndication. In this form, it became widely adopted by Web-log creators (and Web-log software programmers). The blog community saw RSS in this non-semantic, non-summary sense as a way for users to subscribe to ‘the most recent’ postings in their entirely, with the added convenience of aggregating new postings from many different sources. However, nobody knows how much of this subscription traffic is in fact consumed actively. Many users with unmetered broadband are in fact ‘passive’ RSS consumers; they turn on many feed channels with frequent updates (default settings) but hardly ever read any of
Web Information Management
69
the syndicated content piped into their clients. It is just too time-consuming and difficult to overview, yet the user is loath to disconnect for fear of missing something. Bit 3.8 Using RSS to syndicate entire content is a needless threat to Internet bandwidth In 2004, RSS-generated traffic was getting out of hand, effectively becoming a form of network denial-of-service load from some sources. In particular, many clients are poorly implemented and configured, hitting servers frequently and needlessly for material that has not been updated. Adoption of the RDF version for syndicating metadata (RSS 1.0), on the other hand, is an important asset for Semantic Web publishing since it easily enables agent processing. Additionally, it requires far less bandwidth. This kind of publishing and distribution of metadata, with pointers to traditionally stored content, is arguably more important and useful than distributing the actual content in its current format. Syndicated metadata might yet become an important resource to find and ‘mine’ published content in the future Web. At present, however, much RSS traffic is less useful.
Searching and Sifting the Data ‘Mining the Web’ for information is by now an established term, although it might mean different things to different actors. However, we are likely to think of simply harvesting ‘published data’ as-is, without really reflecting on how to compile the results. Pattern matching and lexical classification methods similar to current search engines might suffice for simple needs, but the current lack of machine-readable ‘meaning’ severely limits how automated or advanced such literal filtering processes can be. Nevertheless, much current data mining is still based on simple lexical analysis, or possibly search engine output. A simple example is the way e-mail addresses are harvested from Web sites simply by identifying their distinctive syntactical pattern. The ever-mounting deluge of junk e-mail is ample testament to the method’s use – and unfortunate effectiveness. To accomplish more than just sorting key-value associations based on discrete words and phrases requires a different approach, one that the enhanced infrastructure of the Semantic Web would provide. Perhaps then too a modicum of control may limit abusive harvesting. Usage Mining Broader application of Web mining deals not only with Web data as such, nor even with the metadata relationships, but rather with actual usage. The field of Web Usage Mining (WUM) has attracts significant commercial interest. Although commercial ventures probably account for most innovation and actual deployments in the field these days, their proprietary solutions are rarely published openly. Generally speaking, the visible products (deliverables) in WUM are the finished analysis results and the services to produce them – for example, the Nielsen/NetRatings real-time research and analysis products about Internet users (see www.nielsen-netratings.com). Lists
The Semantic Web
70
of available mining tools and resources are otherwise available at the Knowledge Discovery Nuggets Directory at www.kdnuggets.com – a searchable portal site. The WebSIFT project (Web Site Information Filter, formally the WEBMINER project, see www.cs.umn.edu/Research/websift/) defines open algorithms and builds tools for WUM to provide insights into how visitors use Web sites. Other open sources for WUM theory and relevant algorithms are found in the papers presented at the annual IEEE International Conference on Data Mining (ICDM, see www.cs.uvm.edu/~xwu/icdm.html). WUM processes the information provided by server logs, site hyperlink structures, and the content of the Web site pages. Usage results are often used to support site modification decisions concerning content and structure changes. As an overview, WUM efforts relate to one or more of five application areas: Usage characterization has its focus on studying browser usage, interface-design usability, and site navigation strategies. Application might involve developing techniques to predict user behavior in interactions with the Web and thus prepare appropriate responses. Personalization means tracking and applying user-specified preferences (or assumed contexts) to personalize the ‘Web experience’. It is often seen as the ultimate aim of Webbased applications – and used to tailor products and services, to adapt presentations, or to recommend user actions in given contexts. System improvement means to optimize dynamically server (or network) performance or QoS better to meet actual usage requirements. Application includes tweaking policies on caching, network transmission, load balancing, or data distribution. The analysis might also detect intrusion, fraud, and attempted system break-ins. Site modification means to adapt an individual site’s actual structure and behavior to better fit current usage patterns. Application might make popular pages more accessible, highlight interesting links, connect related pages, and cluster similar documents. Business intelligence includes enterprise Web-log analysis of site and sales related customer usage. For example, IBM offers a number of WUM-based suites as services to other corporations. Most WUM applications use server-side data, and thus produce data specific to a particular site. In cases where multi-site data are desired, some form of client or proxy component is required. Such a component can be as simple as a persistent browser cookie with a unique ID, and having participating sites use page elements to reference a common tracking server that can read it. This profiling method is commonly used with banner ads by advertisers. By contrast, browser toolbars can actively collect profiled usage data from multiple sites directly from the client – for example, feedback to a search engine page ranking function. Many Web users are unaware of browser-deployed WUM strategies such as these. Undisclosed profiling is considered privacy invasion, but it is widely practiced. Bit 3.9
WUM technologies have relevance to adaptive sweb agents
The main difference to current WUM is how the data are used. In sweb agents, most profiling data might never perculate beyond a small circle of agents near the user.
Web Information Management
71
Technologies based on usage mining might come into play when agent software in the Semantic Web is to adapt automatically to shifting user and device contexts. In addition to explicit user-preference data, agent-centric WUM components might mine an assortment of published data and metadata resources to determine suitable configurations and to match offered device capabilities to the tasks at hand. It is reasonable to suppose that profile disclosure to third parties be both user configurable and otherwise anonymized at the agent-Web interface.
Semantic Web Services Query services, Web annotation and other online toolset environments provide simple forms of Web Services (WS), a catch-all term that has received extra prominence in the .NET and Sun ONE era of distributed resources. The development of deployable WS is driven largely by commercial interests, with major players including Microsoft, IBM and BEA, in order to implement simple, firewall-friendly, industry-accepted ways to achieve interoperable services across platforms. WS advocacy should be seen in the context of earlier solutions to distributed services, such as CORBA or DCOM, which were neither easy to implement nor gained broad industry support. The perception has also been that commercial WS development existed in opposition to the more academic Semantic Web (SW). WS developers drew criticism for not respecting the Web’s existing architecture, while the SW project was critiqued for its overly visionary approach, not easily adaptable to current application needs. However, contrary to this perception of opposition, WS and SW are inherently complementary efforts with different agendas, and as such can benefit each other: Web Services can be characterized as concerned with program integration, across application and organizational boundaries. Traditionally, WS handles long-running transactions and often inherits earlier RPC design. The Semantic Web is primarily concerned with future data integration, adding semantic structure, shareable and extensible rule sets, and trust in untrusted networks. It builds on URI relationships between objects, as uniform yet evolvable RDF models of the real world. For example, WS discovery mechanisms are ideally placed to be implemented using sweb technology. A WS-implemented sweb protocol (SOAP) may prove ideal for RDF transfer, remote RDF query and update, and interaction between sweb business rules engines. Therefore, to gain focus we can at least identify a joint domain of Semantic Web Services (SWS). The tersest definition of modern WS is probably ‘a method call that uses XML’, and it is not a bad one, as far as it goes. The W3C defines WS as ‘a software application defined by a URI, whose interfaces and bindings are capable of being defined, described and discovered as XML artifacts’ (as illustrated in Figure 3.1). The current state of support suggests that SWS is an excellent applications integration technology (despite the way many weary IT professionals might cringe at the term ‘integration’), but that it may not yet be ready for general use – important layers in the SWS model are not yet standard, nor available as ready-to-use products.
72
The Semantic Web
Figure 3.1 Web services can be seen as a triangular relationship of publishing, finding, and interacting with a remote service on the Web
The W3C Web Services Workshop, led by IBM and Microsoft, agreed that the WS architecture stack consists of three stack components: Wire is based on the basic protocol layers, XML overlaid by SOAP/XML, with extension layers as required. Description is based on XML Schema, and is overlain with the layers Service Description, Service Capabilities Configuration, Message Sequencing, and Business Process Orchestration. Discovery involves the use of the discovery meta-service, allowing businesses and trading partners to find, discover, and inspect one another in Internet directories. The conceptual stack underlying the WS layer description is expressed in two dimensions. The stack layers run from transport specifications, through description aspects, up to discovery. Descriptions include both business-level and service-level agreements. Overlaying this stack are QoS, Security, and Management aspects. Unfortunately, each vendor, standards organization, or business defines Web Services differently – and in the case of Microsoft WS .NET Architecture, even differently over time. Despite repeated calls for broad interoperability and common standards, some vendors continue to use (or return to) proprietary solutions. The state of WS development is indicated by the general list of resources published and sorted by category at the Directory of Web Services and Semantic Web Resources (at www.wsindex.org). Discovery and Description Two main discovery and description mechanisms are used in WS contexts: UDDI (Universal Description, Discovery, and Integration) and WSDL (Web Services Description Language). They operate at different levels. UDDI is a specification that enables businesses to find and transact with one another dynamically. It functions as a registry interface maintaining only business information, each record describing a business and its services. The registry facilitates discovering other businesses that offer desired services, and integrating with them. Typically, UDDI entries point to WSDL documents for actual service descriptions.
Web Information Management
73
WSDL is an industry standard for describing Web-accessible services (that is, their Web interfaces). As an XML-based language, it is machine processable. However, the intended semantics of the interface elements are only accessible to human readers. In WSDL a service is seen as a collection of network endpoints which operates on messages. A WSDL document has two major parts: first, it specifies an abstract interface of the service, then, an implementation part binds the abstract interface to concrete network protocols and message formats (say SOAP, HTTP). Purpose of WS One way to classify Web Services is by their primary purpose. For example, a few broad application categories are: Administrative Metadata, Agents, Architecture, Brokering, Citation, Collection, Description, E-commerce, Education, Government, Libraries, and Registry. Professionals working within these application areas have increasing access to WS technologies, as these are developed and deployed, but only to the degree that they become aware of them. An example might illustrate the issues and directions. Efforts related to the management of metadata currently tend to focus on the standards and tools required by applications within the group’s main constituency. An early primary focus is e-license management for digital libraries. Librarians everywhere have increasingly been struggling with the management of electronic resources, but they have not known where to go for help in an area of the profession that seemed devoid of expert advice. For example, expenditures for electronic resources have grown enormously in the past decade, accounting for upwards of half the library budget in some cases. Yet the ability of library staff to manage this growing collection of resources typically remains unchanged; most of the time is still spent managing print resources. Commonly used integrated library systems do not provide tools to help manage electronic resources. Administrative metadata, elements about licensed resources, do not fit comfortably into most library systems. It is difficult for libraries to know what use restrictions might apply to e-journals and aggregated online texts, and how to triage and track related access problems. Most libraries simply lack the new kinds of tools required by these processes. Only in recent years have Web-based resources became available and been published in any greater way. One such resource, useful for tracking developments in this field, is the Cornell University ‘Web hub for Developing Administrative Metadata for Electronic Resource Management’ (at www.library.cornell.edu/cts/elicensestudy/home.html). The Web Hub is especially useful in highlighting e-resource management solutions implemented at the local level by different libraries. Such systems are typically developed by large research institutions, perhaps affiliated with a university library, but a growing number of vendors seem interested in developing generic products to meet the needs of most academic libraries. Whither WS? The entire field of Web Services remains in a state of flux despite several years of intense development, and not all proposed ‘Web services’ have very much to do with the Semantic
74
The Semantic Web
Web. In part, there has been a trend to call every new incarnation of e-commerce functionality an e-service, or if on the Web, a WS. In part, some confusion has been due to vacillation on the part of Microsoft (MS) in both its definition and deployment of .NET component technologies. Confusion and caution reign. On the one hand, there is the risk that the MS approach would fragment the Web into incompatible WS segments, simply due to the massive dominance by MS in the desktop market. On the other, there has been the fear that the new Web would drift into the embrace-and-extend sphere of proprietary-modified open standards, thus defaulting into a kind of corporate hegemony control of protocol and security by a single company. It is a fact that core MS WS runs on proprietary code only on MS Windows servers. A number of independent projects have been progressing under the guidance of several major actors. We will now look briefly at the major WS efforts, ending with Microsoft’s .NET. A realistic view might be that, ultimately, a successful deployment of SWS as a new Web infrastructure will require that all the actors who are committed and cooperating around open standards should achieve the vision, or at least interoperate with it. At present, however, the major players are in critical areas still (or again) pushing proprietary solutions, which risks locking corporate environments into vendor-specific commitments difficult to migrate from later. Fortunately, the WS paradigm is firmly message based, depending on XML, SOAP (or any other XMLP-compliant protocol), and other platform-independent specifications. This independence gives hope for bridging vendor divides. The Sun Approach Sun Microsystems promotes the J2EE platform for WS (java.sun.com/j2ee/), through the intermediary of the open Java Community Process (JCP, www.jcp.org). Both standards and architecture are open and distributed. The platform is based on Java, in theory independent of operating systems. As Sun invented the Java language and promotes competitor Unix-based operating systems, WS-platform debates often become slanted as a ‘Sun-interests against MSinterests’ confrontation. Such acrimony has often been detrimental to Web-universal solutions. For developers working in Java, the announcement in 2002 of an official suite of API specifications for Web services from Sun was welcome news. Sun had long been criticized for inaction on this front, given the early availability of the MS .NET SDK. However, rather than proposing transitional hack solutions to as-yet uncharted technical areas, Sun’s strategy had been instead to work with open source initiatives on the major pieces of the Web services puzzle. Also announced was the Java Web Services Developer Pack (JWSDP) to provide an integrated development platform. In the JWSDP, Sun packaged some reference technology implementations into a single, free download, adding some infrastructure to ease Web service deployment. Thus, the JWSDP let Java developers create useful Web services with available components, instead of forcing them to either assemble those tools themselves or
Web Information Management
75
use another vendor’s tools. Importantly, it is all open source, not proprietary code, and anchored in a developer community. Other aided tasks include compiling and registering a WSDL file for the implemented Web service, so that others can find it online and use it. Supported in this context is the UDDI v2.0 registry standard. Metering and monitoring capability as of JXTA v2.1 enables automated query and inspection of traffic and memory usage on remote network peers. Using Java makes core WS-implementations platform independent, though at the apparent cost of committing to a single language. In actuality, support for other languages does exist, though Java remains the primary language for implementations. Java is good for building complex systems but has itself been perceived as complex. Recent technology releases have positioned developers in a better position to address the challenges of building real-world Web services. Apache Group The Apache Group collects a number of evolving WS efforts at ws.apache.org. Since the group is historically defined by the web server, it is natural that several WS projects evolved here. Axis (ws.apache.org/axis/ ) is an implementation of SOAP vl.l, the proposed standard fulfilling the W3C XMLP requirements (see the protocol overview in Chapter 5). Essentially, it corresponds to what would have been Apache SOAP v3.0, a previous project, but rewritten from scratch. The project has focus on the basic building blocks for implementing WS primitives. Since Apache is a major server workhorse on the Web, having native SOAP support goes a long way to implementing WS applications on existing Apache servers, perhaps best expressed in the Tomcat and WebSphere application server contexts. Axis builds on the Apache notion of pluggable modules and APIs to extend core server functionality with advanced message processing and request forwarding to defined services. Other projects of interest include: WSIF, a simple Java API (invocation framework) for transparently invoking Web services irrespective of their location. WSIL, a distributed Web service discovery method (inspection language) specifying how to inspect a Web site for available Web services. Several other projects can also be found that are considered more experimental. DARPA DAML WS The DAML Services arm (www.daml.org/services/) of the DAML program (DARPA Agent Markup Language, see the DARPA section in Chapter 7) started by developing a DAMLbased Web Service Ontology (DAML-S), with early releases of specifications in 2000–2003. The DAML focus is mainly on the languages, specifications, and protocols involved in the client-WS communications, defining the common interfaces that should be shared by technologies developed by others.
76
The Semantic Web
In a subtle refinement of focus in November 2003, however, the renamed Semantic Web Services arm released the expected language implementation specifications as OWL-S v1.0 instead. OWL (Web Ontology Language, described in Chapter 6) received official W3C recommendation in August 2003, and this move toward implementation status has naturally affected other projects. The current OWL-S program also works on supporting tools and agent technology to enable automation of services on the Semantic Web. The focus is on descriptions of WS at the application level, describing what a particular WS implementation does, rather than specifying how it does it. Other WS description languages, such as WSDL, are specifications at the communications level of message and protocol, while discovery mechanisms such as UDDI serve mainly as pointers to service addresses and ports. DAML-S/OWL-S complements the lower-level specifications prevalent in these other approaches that are mainly concerned with the core platform architecture. DAML-S/OWL-S deals with the application-level issues of discovery, invocation, interoperation, composition (of new services from existing selections), verification, and execution monitoring. A service implementation must in this context specify two things: A service profile to describe what it requires from agents and what it provides for them. A service model (known as a Process) to describe workflow and possible execution paths. A profile would be used both by the WS implementation to advertise its availability and capabilities, and by requestors to specify their needs and expectations. Brokering services can then match requestors and providers based on their respective profiles. A third subontology may specify how service details are implemented (Grounding). Despite having generated a lot of interest through its promise to add semantics to WS descriptions, only a small number of these WS descriptions appear as yet to exist. Furthermore, most of these do not point to real services, representing instead constructs from the DAML-S community. A possible cause may be the perceived lack of sustained contact with and interest from the main implementers driving deployed WS-technology, Microsoft and Sun, both of whom have vested interests in promoting in-house WS languages and specifications. A 2003 Dutch study of the situation, ‘An experience report on using DAML-S’ (www.iids.org/publications/essw03.pdf, also see www.iids.org/pubs), concluded that DAML-S was superior to existing WS languages as it allowed the use of formally defined domain knowledge. However, it did not yet adequately address real-life situations. With regard to the following years of OWL-S development (v1.1 was released in November 2004), much the same concerns and conclusions appear applicable. Effort is being made to integrate WS made up of many different services into a semantic-interoperable context for practical business use. The DAML site references tools, security issues, example services, and a number of ontology/agent use cases, sorted by architecture and by language. Environment categories studied include B2B, Grid, Ubiquitous, Web, and Agent. The language-oriented categories
Web Information Management
77
include Advertising and Matchmaking, Negotiation and Contracting, Process Modelling, Process Enactment, and a catch-all Unclassified. Recognizing that information security and privacy play an increasingly important role in a world in which the Web is central to business, industry, finance, education, and government, several OWL-S categories have been proposed and developed to resolve such issues: Credential Ontology defines the capability to specify access control restrictions of Web pages or Web services that use authentication as a requirement for authorized access. Security Mechanisms Ontology defines the capability to interface on a high level of abstraction among various security standards and notations. Service Security Extensions Ontology defines the capability to annotate the security properties of SWS. Agent Security Extensions Ontology defines the capability to annotate the security properties of agents. Privacy Ontology defines the capability to express privacy policies to protect information, and a protocol to support matching of privacy policies across different contexts. Cryptographically Annotated Information Object Ontology defines the capability to capture encrypted or signed input or output data of services. One may see these efforts as a measure of technology maturity on the road to implementing practical services to realize the greater sweb vision. .NET Web Services Assessing the .NET (‘dot-net’) way of implementing an ‘intelligent Web’ and its related services proved difficult. Where the previous constellations of effort, even at Sun, had clear focus on technical implementation and open discussion among developers, with Microsoft one entered the realm of corporate marketing strategies and largely proprietary technology. Whatever the expressed (and shifting) aims of the .NET effort, the dominance of MS applications on the desktop and in the corporate intranet (and the way these applications work) also ensures that any Microsoft policy to ‘embrace and extend’ existing open or industry standards inevitably affects deployment of sweb technologies. Current MS overviews (for example at www.microsoft.com/net/ and msdn.microsoft.com/ webservices/) describe mainly XML Web Services. These XML-WS are somewhat loosely defined – in business terms by promised synergies, and in technology terms by selected client-server capabilities: Partner and client connectivity through exposure of useful functionality over an ‘industrystandard’ Web protocol (mainly SOAP over HTTP). Personal, integrated user experiences through smart devices that utilize sufficiently detailed descriptions of WS interfaces to allow client applications to talk to the services (based on metadata published as WSDL). Increasing revenue streams by enabling businesses to create their own WS, advertised through registry discovery so that potential users can find appropriate services easily (using UDDI).
78
The Semantic Web
Security aspects inherent in using open HTTP and site-dependent authentication systems are currently resolved by extending SOAP with security components, following the published WS-Security Specifications. Other, more secure transports are promised as soon as the technology matures and higher-level services are developed. The official view is regularly restated that Web Services ‘have made a lot of progress’ over the last few years, and that work on the platform continues. However, the long status documents from Microsoft (see www.microsoft.com/net/basics/) remain remarkably vague about practical applications, even after several years. The reader may consider it fortunate that although Microsoft has a long-standing bad habit of sowing serious linkrot by often and gratuitously ‘reorganizing’ its extensive Web sites, in recent years it does implement a modicum of redirection from older URLs, at least for a time. The previous URL paths have if fact ‘only’ suffered changes to mixed case. One functional service is Microsoft Alerts (www.microsoft.com/net/services/alerts/), by now fairly established. The original ‘.NET’ moniker in the name was dropped at some point. Alerts is a notification service that delivers personalized news, weather, and information to people on the move – in effect, a streaming-content aggregator service using RSS. It is hard to assess how pervasively used it is, but it has been promoted heavily for several years and integrates with the typical suite of Windows productivity applications. Other staple services offered (such as Passport, Messenger Connect for Enterprises, and MapPoint) are not ‘complete’ but are components requiring integration with larger business processes. The view expressed as early as 2001 remains curiously unfulfilled, despite the constant hype, that by simply subscribing to a rich selection of available ‘plug and play’ WS functions, developers could create a plethora of ‘richer user experiences’ quickly and easily. Larger-scale deployment of .NET services appear mainly as restricted trials within Microsoft itself, as described in a series of whitepaper studies. Development focus at Microsoft has been on ‘management services’ such as download center, content storage and retrieval, and generic Web platforms. Less seriously, Microsoft employees developed an internal service to track left-over snacks, SnackAlerts, which can notify subscribing employees where the snacks are in 79 Microsoft buildings around the world. Microsoft’s WS specifications are a moving target, evolving in many places in parallel – an ongoing process where they are defined, refined, and sometimes deprecated seeming at whim. Neither are we sure of who the major players are, or will be, since it depends on the practical scope of interoperability and chosen authentication mechanisms – how open MS chooses to make them. Therefore, it is impossible to say with any great precision what .NET is, or will be, until the respective applications are ‘fully deployed’ and functional. It is not enough to say it ‘is a set of software technologies for connecting information, people, systems, and devices’, yet many MS explanations say little more. Even the promise of ‘small, reusable applications’ in XML for WS seems to ignore the complications of real-world business applications. Perhaps the reason is that most of the implementations are buried in corporate systems.
Web Information Management
79
Bit 3.10 Visible MS XML-WS appears mainly to be a commoditized side-effect of selling Web-aware solutions to implement core corporate business processes This considered view is based on the fact that MS has long had only two core business models, supporting corporate solutions and selling the ubiquitous Windows desktop, combined with a proven track record of successful salesmanship based on the concept of convenience. Some amusement has been occasioned in the press over the fact that even lead spokespeople for Microsoft have had great difficulty explaining the ‘.NET vision’ intelligently or consistently. Developers in the .NET area might reasonably take issue with the apparent dismissal suggested by this critique, engrossed as they are in specific technical details that appear well-defined from their insider perspective, but many analysts agree that the overall MS strategy is muddled at best. Bit 3.11
Sweb WS is mainly an architecture model rather than a market sector
The Microsoft approach is typically characterized by a product focus on componentapplication-suite. Therefore, management appears genuinely confused in pursuing the ‘Web Services market’ and by how to transition from vision to product. Stated goals slide into obscurity each year as the next great initiative is announced. Is that any way to run a vision? One critical factor is that the design of MS software has often had lock-in effects on ensuing versions of hardware, making alternative solutions difficult. Consider how many companies are locked to the MS-Word document format(s) despite attempts to develop format-compatible third-party software. Many aspects of current .NET WS implementation require proprietary MS software at some level or another, possibly with lock-in features to specific hardware. Of particular concern is the introduction of a proprietary ‘information rights management’ (IRM) mechanism in MS Office 2003 and later applications. This implementation of IRM can block general access to Web-published documents produced using them. At the very least, it may prove legally impossible to use third-party applications to view such documents. Either way, it breaks the established concept of universal access to Web-published content. We cannot deny that there are legitimate reasons for corporate users (and even individuals) to desire effective content-control mechanisms. However, once deployed, it will be awkward and rare to produce any content without inadvertently ‘locking’ it for other applications and users. Bit 3.12
Current IRM/DRM mechanisms are a threat to the Semantic Web vision
When the dust finally settles, ways may have been found to let free exchange of information on the Web coexist with economic interests. Non-intrusive micropayment solutions and less draconian copyright and licensing enforcement appear to be necessary ingredients for such a resolution.
80
The Semantic Web
A mitigating circumstance is the recently adopted policy of the W3C to recommend particular technologies for the Web only if they are freely offered for the public good without restrictive ties. Although this policy ensures open standards in the infrastructure and WS, it does little to stem user adoption of needlessly restrictive technologies when producing and publishing Web content. DotGNU The DotGNU project (www.dotgnu.org) was started in reaction to, and aims to be a complete and open-source replacement for, Microsoft’s .NET initiative. It independently implements the Common Language Infrastructure (CLI) that defines ‘.NET’ functionality. Revisited in 2005, the project has evidently gained considerable adoption, claiming to be the industry leader and provider of Free Software solutions for desktop and server applications. Three major components of this initiative are currently being implemented: DotGNU Portable .NET suite of tools to build and execute .NET-compatible applications on any platform. The cross-platform suite includes, among other things, C/C# compilers, assembler, disassembler, and runtime engine. PhpGroupWare suite to provide multi-user collaborative Web-space, including many useful XML-RPC Web-service components ready to use in custom WS implementations. DotGNU Execution Environment (DGEE) to provide virtual server implementations and resource management support to run the WS implementations. The project goals are to provide a reasonably .NET-compatible system and then improve on what Microsoft is offering – for example, by decentralizing it. Therefore, DotGNU is based on a peer-to-peer system for service discovery, avoiding the use of centralized authentication/authorization portals such as the ‘Passport’ system. The strategy for DotGNU is not only technologically superior, it is claimed, but it also serves to ensure that no single entity would ever have the kind of monopoly power to control (or end) the free exchange of digital information. The latter is openly presented as a strong motivation: . . . to prevent Microsoft from achieving their stated goal ‘that the era of ‘‘open computing’’, the free exchange of digital information that has defined the personal computer industry, is ending’. Read the quotation in context and learn how the Free Software movement works to prevent this disaster. The contribution of the DotGNU project might turn out to be important.
In keeping with the open approach, the developer community runs DotGNU Wiki (wiki. dotgnu.org) as its forum of discussion and general Web resource. Using WS At this point, you might well wonder how to use Web services when the platforms are still evolving, and when there is so much flux, uncertainty, and lurking incompatibility. In fact, today’s Web service tools and frameworks, irrespective of vendor, do provide enough core functionality to build interesting distributed applications. Expect a steep learning
Web Information Management
81
curve, however, and be prepared to develop your own solutions, or to pay for proprietary solutions developed by others. Some higher-level services such as WS-Security are beginning to gel with the advent of support in a number of new tools (such as the Web Services Development Kit). Other services are still in their development stage as specifications are reviewed and early implementations expose areas where the specifications may need solidifying. Until then, you can rely on traditional HTTP-level mechanisms for security and other features. From the business perspective, some make the analogy that Web Services are similar to business services. The comparison is to a great degree a valid one, but at the same time it is constrained by the products and services view. However, just as most companies outsource aspects of their business process that are not their core competencies, such as shipping and materials handling, Web services might correspond to outsourced development of certain services they use all the time but may not wish to implement in-house. Authentication is the one such service implemented and deployed in a major way on the public Web. A substantial number of Web sites today rely on the .NET Passport authentication services, even though it has both suffered numerous embarrassments and received considerable criticism. The service is required when using any other .NET WS implementation. Originally introduced with an absolute dependency on Microsoft central servers, the current .NET approach appears to allow at least a limited delegation of authentication authority to other non-MS authentication networks. In late 2002, this security interoperability was mediated between MS and a large coalition of independents under Liberty Alliance (www.projectliberty.org) by MS-partner and LAfounding member RSA Security (www.rsasecurity.com). RSA, with its critical focus on Secure Web Services, and a deeper understanding of the positions on both sides of the divide, clearly had a pivotal role here.
Security and Trust Issues Reliability, availability, and security of data and services are goals that sometimes seem difficult to attain even under centralized control and governance. How then can one possibly realize them under the distributed and ad hoc management that characterizes in the Semantic Web, with its mix of roaming users and authority-delegated software agents? How indeed? Part of the answer depends critically on the adopted solutions for trust, discussed later. Central to the issue is a pervasive and transparent management of strong public key cryptography, especially in the form of digital signatures, complemented by webs of trust and proofing logic. This same technology can be used, not only to verify reliably unique person identities, even anonymous ones (!), but also to identify uniquely and make tamper-proof particular documents, document versions, and in fact any Web resources. It is for this reason that the visual map of the Semantic Web (Figure 2.5 in Chapter 2) includes encryption, signature, and proof logic as key parallel components.
The Semantic Web
82
Bit 3.13
Authenticated authority must work in a roaming context
As one of the cornerstones in the Semantic Web context is that access should be automatic and transparent regardless of location or used devices, the underlying authentication model must remain valid and secure across many ad hoc platforms. These same infrastructure components can also resolve some of the problems facing attempts to implement reasonable business models in a Web context. Technologies may change, but the essential requirements remain much the same, comprising the key concepts of Authentication, Authorization, Integrity, Signature, Confidentiality, Privacy, and more recently, Digital Rights Management and Information Rights management. Table 3.1 summarizes the meaning of these concepts. Another essential requirement of security standards in the Web context is that they work naturally with content created using XML (or with XML-derived languages and protocols). Transparency is another essential characteristic, in that integrity, confidentiality, and other security benefits should apply to XML resources without preventing further processing by standard XML tools – whether at message endpoints, or in intermediate processing. Although older security technologies provide a core set of security algorithms and technologies that can be used in XML contexts, the actual implementation of these is inappropriate for most XML applications: The proprietary binary formats require specialized software for interpretation and use, even just to extract portions of the security information. Older technologies tend to assume specific components integrated into the endpoint applications, introducing awkward dependencies. Older standards are not designed for use with XML and thus lack support for common XML techniques for managing content (such as URI or XPath pointers). Table 3.1
Summary of essential security concepts
Concept
Question answered
Comments
Authentication Authority Integrity
Who am I? (Verify asserted identity against some trusted authority.) What may I access and do? Is the information intact?
Signature
Is the information certified?
Confidentiality
Is the information safe from unauthorized disclosure?
Privacy
Is individual and sensitive information safe from unauthorized disclosure? How may I use or share this information?
The later Trust section further discusses authentication, identity, and role issues. Individual and role. See above. Prevent accidental or malicious change, or at least detect it. Ties in with identity issues. Might certify an identity or authority. Encryption makes information unreadable even if access controls are circumvented. Governance issues of how to use sensitive information. Consent.
Digital Rights Management
Usually now a combination of access control and embedded enforcement of usage license.
Web Information Management
83
A unified and open framework of new sweb-oriented standards and implementations, however, is evolving to address these issues on the Web. Since public and corporate awareness seems dominated by a focus on Microsoft’s .NET solutions, it is important to explore the subject in general terms. In particular, we need to highlight the alternatives to central authentication by proxy authorities, with their proprietary interpretations of ‘trust’ and ‘security’ as a ‘product’ to sell, in order to assess properly the role of these concepts in the broader Semantic Web context.
XML Security XML Security defines a common framework with processing rules that can be shared across applications using common tools. Therefore, no extensive customizing of applications is needed to add security. While XML Security reuses the concepts, algorithms, and core technologies of the legacy security systems in the interests of interoperability, it also introduces the changes necessary to support extensible integration with XML. The core XML Security standards comprise two Semantic Web cornerstones seen in the map and stack views (Figures 2.5 and 2.6) introduced at the end of Chapter 2: integrity and signatures, as XML Digital Signature (XML DigSig); confidentiality, as XML Encryption (XML Enc). Persistent integrity and security is associated with the content itself, rather than with any single transport, processing, or storage method. The implementation thus provides seamless end-to-end security, irrespective of intermediate processing, and reusing existing technologies when possible. XML Security realizes these goals by the simple expedient of transforming binary representations (such as for digital keys, certificates, or encrypted data) into plain text – or rather to a well-defined subset of text (such as Base64) that is guaranteed invariant in other message processing on any platform. Residual issues do remain in the conflict between exact representation in signed text and digests, for example, and the relative laxness in XML that allows whitespace variations and some token substitutions (such as single and double quotes) in ‘identical’ messages. Such issues are generally resolved by transforming to ‘canonical’ form before applying a signature. Related supporting core standards not explicitly on the simple sweb map are also needed: Key Management, as XML Key Management Specification (XKMS), defines protocols for the public-key management services and identity binding needed to implement data integrity and signatures. Authentication and Authorization Assertions, as Security Assertion Markup Language (SAML), define authentication and authorization assertions and shared framework used to establish identity and authority. Authorization Rules, as XML Access Control Markup Language (XACML), state the rules and policies used to manage access.
The Semantic Web
84
In addition to these standards, we must add major XML Security applications: Web Services Security, as WS-Security specifications that outline how XML DigSig and XML Enc may be used with XMLP (SOAP) messages. Also specified is how security claims (for example, identity credentials) may be included with a message. Privacy Management, as specified in the Platform for Privacy Preferences (P3P), defines the XML vocabulary to manage human-policy issues ensuring that organizations and individuals with legitimate access to information collected about an individual do not misuse it. Digital Rights Management, in the form of extensible Rights Markup Language 2.0 (XrML), provides a framework, a language and processing rules to enable managed interoperability in content workflow with multiple participants and applications, and among different ‘viewers’. We might note in the context of WS-Security that Microsoft and IBM released a Web Services security architecture and roadmap in 2002 that outlined a strategy and series of specifications to bring together different security technologies. Despite the stated aim of providing a unified, flexible, and extensible security framework for WS-Security, the core group of IBM, Microsoft, Sun, and VeriSign subsequently started pushing proprietary solutions instead.
Trust The concept of trust has both formal and informal components, but in both cases ultimately it is based on accepted knowledge, such as an established identity. In the more formal contexts, this knowledge requires some form of explicit verification; in the informal ones, it is implicit (typically in a shared history). Bit 3.14
Both sweb agency and e-commerce require secure authentication
Whether implemented authentication mechanisms become tightly centralized or distributed trust systems remains to be seen. Perhaps a middle ground can be found. In this first discussion of trust, the focus is on the fundamental issue of authenticating identity assertions. Establishing trust rests on some process of authentication, and on some ultimate authority that can validate the assertion about a given identity. Opinion diverges about whether central authorities or distributed ones are most appropriate (or secure) for the Web. Using a central authentication proxy raises many concerns, from scaleability and technical reliability of systems not yet tested on such a large scale, to entrusting most matters of personal identity to a single arbitrator with unproven security and unknown agenda. Simply rolling out a single identity-authentication model devised for corporate use and expecting it to scale and function on the general Web for everyone in all situations seems foolhardy at best.
Web Information Management
85
Instead most sweb perspectives assume distributed systems that can if needed interact. Successful authentication results in some form of credentials (or certificates of trust, physical or electronic) that the user can present to querying parties to be granted authorization for intended actions. Implicit in this context is the requirement that the credentials are acceptable to a querying party (often custodians for protected data or access), and that lowoverhead mechanisms exist to validate on demand the authenticity of the presented credentials. Although, typically, we refer to ‘identity’ as if it always implied the concept of unique personal identity, a trusted identity can have other faces. The simplest real-world example is delegated authority, which means that the bearer of some issued credential becomes a proxy for the identity and is conferred with some measure of authority to act on the behalf of this person. Depending on circumstances, generally the degree of value at stake or ‘risk’ involved, the bearer may be accepted at face value (possession is sufficient), or in turn be required to authenticate and validate against for instance a class of permitted bearers. As implied elsewhere in the context of sweb agents, delegated authority is expected to be the norm rather than the exception on the Web. Therefore, we had better get the whole identity-authentication model right when implemented in order to support securely this functionality. Table 3.2 exemplifies some alternatives to personal identity. In no case is the trust system’s knowledge of identity absolute. Instead, it is based on various situation-appropriate criteria for an ‘acceptable’ level of confidence that an asserted identity or type of access is correct, a level that can vary greatly between different contexts. Table 3.2 Different identity concepts considered in authentication situations Identity type
Question answered
Comments
Personal identity Knowledge identity
Who am I to be trusted? Do I know something that corroborates my asserted identity? Am I the same person as in a previous authenticated context? Am I just here and now entrusted to perform a particular action? Do I belong to a trusted group with authority in this situation? What am I, as part of a role group, allowed to do in a given context? Am I seeking access in the appropriate context?
The basic identity query. The basic PIN- or password-challenge situation.
Consistent identity Purpose identity Group identity Role identity Context identity
Presence identity
Am I in the correct location to be trusted in this context?
Enables authentication of anonymous yet trusted identity. In other words, the presented ‘key’ is trusted in context on its own merit, not any specific bearer. Simplifies the management of access rights and operation privileges. As above, and often combined with it. Can involve prior authentications, identification of associated software or hardware, trusted protocols, tracking mechanisms, or external methods. May also include the issue of request occurring at appropriate time.
86
Bit 3.15
The Semantic Web
Signatures enable verifiable trust referrers and testimonials
Essentially, an unknown Web service can assert: ‘Trust us because of the credentials of the following named individuals who vouch for us’. Digital signatures allow others to confirm the veracity of asserted identities and credentials. In many authentication and certificate contexts today, the primary focus is on the personal identity of the user only in the context of a particular system, and hence on local mechanisms for the user to prove the asserted identity – typically, account name and password. Any context that requires ‘global’ scope must satisfy higher demands: Uniqueness of identity. An e-mail address may therefore be better than a name or number, as it leverages an already globally unique Web URI that is personal and human readable. Portability. Unless specific security requirements dictate otherwise, it should not matter from where or over which media the access request comes. Ideally, the authentication dialog should occur encrypted, but there are perhaps situations where it is not possible. Multi-factor components. Realistic security can often be achieved by combining (hashing) several different and independent validation factors, such as asserted identity, PIN/ password, and certificate. One or more factors may be varied at random by the server. Ease of authentication. It is well-known that users will invariably find workarounds to defeat ‘difficult’ authentication measures. Yet ease of use should not compromise security. Adequate safeguards. The system must strike a reasonable balance to provide an acceptable level of confidence in the authenticated identity and the integrity of the subsequent session. Table 3.3 summarizes some of the specific security issues raised, along with possible solutions. Authentication of identity (and subsequent authorization) can be accomplished by any one of the following processes, in increasing degrees of formal security span: Non-existent. The system accepts your assertion at face value. For example, messaging or chat systems often require only a session-unique but arbitrary handle for each user and particular context. Underlying account identity is authenticated only by password sent in clear, which is generally stored in the context of the client system and sent automatically. Implicit. Authorization may be based on physical access to an identifiable instance of client hardware or software, or possibly to a stand-alone device that generates one-off certificates. Authentication is then assumed satisfied by external or informal processes that regulate such local access or device possession. Derived. Based on previous successful authentication (local or Web). The reasoning applied is that trusted users in one context are automatically trusted in another on the same network. Most intranets and many Web/sweb contexts support ‘forwarded’ authentication. Explicit. Authentication mediated by a remote server/service with well-regulated centralized authorization processes, exemplified by network login, typically with many access levels. External. Secure authentication mediated by trusted service (a central or distributed authentication service that issues encrypted certificates).
Web Information Management
87
Table 3.3 Summary of authentication issues in different contexts Identity type
Issues
Comments
Personal identity
Some systems and contexts might not care about more than local uniqueness of name handle. Typically combined in the form identity-name plus PIN/password.
Consistent identity
Trust in certifying authority. Depth of authenticity check. Uniqueness. User-security of known credential. Criteria for selection. System management. Certification method.
Purpose identity Group identity
Similar to a physical key. Shared access.
Role identity Context identity
Usually shared. Potentially shared.
Presence identity
Not always valid for user. Personal integrity issues.
Knowledge identity
Public key can prove consistency of secret key holder, directly or through key-generated hash of other data. Authority invested in the bearer. Often combined with personal identity. Often synonymous with Group. Less often used, rarely alone. Must be adequately defined. May be related to context.
While still rare, network or external ‘public’ authentication based on digital certificates is being deployed in newer solutions with increasing frequency. This solution might tip the balance towards the requirement always to use such strong authentication, even in casual Web client use. The .NET Identity Model The proposed .NET model of identity is a layered concept of personal identity, at minimum a three-tier structure that uniquely specifies the individual, the application that the individual is running, and the location where this software is running. Instead of an application being the gateway to the desired data, the model sees the user as the gateway to the data. This shift in focus is good and well, as far as it goes; the particular application used should be incidental. However, for reasons evidently more to do with product licensing than user convenience, or Web functionality, the .NET identity model is unfortunately strongly tied to device and location – two aspects that should be transparent and largely irrelevant in the Semantic Web. Bit 3.16
Identity on the Semantic Web should not be needlessly fragmented
A sweb-aware person is likely to use a multitude of devices from a multitude of locations. To deploy an authentication infrastructure that ties identity to particular instances of these two variables seems, to put it mildly, counter-productive. It is as if your legal name changed depending on whether you were home, at work, or in your car.
88
The Semantic Web
The .NET approach has other weaknesses – perhaps the greatest being the way the model tries to apply a ‘one size fits all’ authentication process for all situations, serious or trivial. Access to any Web service here requires a centrally issued certificate for each session. The system presumes a single authority for global trust, regardless of the session context. Perhaps more to the point, the proposed model is unproven at the scale required, and the trustworthiness of the system not universally accepted. Different Trust Models Alternative, distributed trust models have the benefit of corresponding trust structures already functional in the real world. For example, there are several different possible authentication models, modelled on corresponding ones in the physical world: Single central authority, which is/was the .NET model, with all user authentication certified by one instance and database. We have no single central authority for identity in the physical world to compare with, but the national identity registries in some countries come closest. Independent regional authorities, which we recognize from the physical world in identity credentials issued by each national government for its citizens. Such credentials are generally accepted on sight within the country without explicit validation, and usually also abroad (at least in ‘internationalized’ versions, such as a passport). The latter acceptance is due to international consensus to trust implicitly national authorities to define citizen identity. Delegated authorities, exemplified in the real world by various issuing authorities for public and special-purpose ID cards. Although the identity ultimately derives from national identity authentication, delegated certificates are used in their own right. Acceptance of such credentials is usually limited to entities within the country of issue. Purpose-oriented authentication services, which correspond in the real world to institution-issued or corporate certificates of identity or purpose. Acceptance for corporate-based credentials varies greatly, and these credentials are valid outside the nation or region domain in only a few cases – such as when due to organization alliances, such as VISA for bank-issued credit cards. A Web example is the system of Digital Certificate chains that emanate from some root Certification Authority (CA). Distributed authentication services, which are useful for generic identity validation in any context where it is considered adequate – perhaps only to prove ‘the same identity’ across sessions. This model is perhaps best exemplified in a working way by the distributed ‘webs of trust’ made possible by public key cryptography – PGP key server networks. Community authentication, which boils down to being vouched for by people who know you or trust your assertions. This level of personal authentication is informal but practical, and is historically the common basis for human trust systems in general. It is leveraged into the digital world of the Web by PGP certificates and digital signatures. Not explicitly factored into these model descriptions is the issue of trust history, known as reputability. For example, you are personally more likely to trust a person or organization on the basis of successful past dealings (that is, private reputability) than an unknown.
Web Information Management
89
An established community reputation (or public reputability) is a valuable but intangible asset for brick-and-mortar businesses. However, it proves hard to establish a comparable asset in the realm of online Web business, at least without broad brand recognition. Business reputation may be assessed by interpreting formal records such as registration, tax history, share performance, and published analysis, but such investigation and evaluation is nontrivial. Bit 3.17
A trusted identity can be verifiable yet remain essentially unknown
Authentication of identity in the Semantic Web can be resolved adequately in many situations by just being able to prove the consistency of a verifiably ‘the-same’ identity across different contexts, not necessarily an absolute identity. Table 3.4 summarizes the previous trust models and the issues relating to each. The difference between models is where the line of trust is drawn and the how far that trust reaches. Who do you want to trust today? In all cases, there is an ultimate point in the recursive validation of authenticity beyond which the querying party must simply take the identity assertation at face value, on trust of the issuing authority. Trust is not only about identity, but also about whether published content or provided services are trustworthy or can be validated. While many content-trust assessments are Table 3.4 Summary of authentication models and related trust issues Model
Problem
Possible solution
Central authority
Single source of vulnerability. Too much central control. Heavy demands on services. Varying or unknown degree of trustworthiness. Validation difficulties. Varying or unknown degree of trustworthiness. Limited scope. Validation difficulties.
Delegation. Alliance around common standards.
Independent regional authorities Delegated authorities
Purpose-oriented authentication
Might be ‘trust the bearer’ style. Varying or unknown degree of trustworthiness.
Distributed authentication services
Varying or unknown degree of trustworthiness. Validation difficulties. Too little control. Circular chains of little span. Who to trust. Reputability.
Community authentication Self-proclaimed
Trust on what basis?
Alliance of trust. Common minimum criteria for authenticity. Common open standards. Trust chains. Vetting through central or regional clearing. Broad alliances of acceptance. Used only within purpose domain. Combine with confirmation codes (such as PIN) and two-component login. Webs of trust. Common open standards. Promote broad alliances of acceptance. Leveraging trust in community as such. Real-world analogy. Reputation influences trust. Known history. Moderators. Peer voting (auctions, supplier comparisons). May evolve into community trust.
The Semantic Web
90
typically made with explicit or implicit reference to identity-trust validation, more generic models are required for semantic processing. Trusting the Resources Not only do we need to deal with trusted and untrusted identities, we also need to assess the trustworthiness of the information resources and services themselves. The issue is especially relevant to library-information application contexts that need to validate content ‘truth’ and dependencies. A growing research area deals directly with knowledge provenance and the development of technologies to identify, categorize, and process uncertain data automatically. Bit 3.18
The Web will always offer uncertain and incomplete information
It is in the nature of the anarchy, so to speak. However, it should be possible to annotate Web content to at least create islands of local certainty. Provenance level can range from strong (corresponding to high certainty and trust), to weak (that is, low certainty). The strongest category, static, concerns assertions (so-called ‘facts’) based on ‘propositions’ that do not change over time and can thus be validated once. A weaker level of provenance, dynamic, concerns changing data, where validity might vary over time. Weaker still on the provenance scale are ‘uncertain’ data, where even the ability to validate the current data is uncertain. The lowest level of provenance is context-relative, and typically called judgement-based since ‘truth’ then depends on human social processes and context-dependent human interpretations. Legal knowledge is one example of this category. Although most real-world ‘facts’ are at least potentially variable and context-dependent, we can often treat them as ‘static’ for specific purposes and intervals, perhaps defined with formal constraints. This simplification makes the issue of provenance and validation easier, even though assessment must often rest on numerous implicit assumptions. Bit 3.19
The Web will always offer undated and outdated information
The amount of undated information on the Web is surprising, even when researching academic writings. Sometimes, the only way to determine the topical relevance of a given source is through a careful study of the cited external references and that way make a reasonable assessment of ‘not written before and not later than’ dating interval. For static provenance processing, we need to answer three questions: What is the truth value of any given proposition? (True, false, or uncertain.) Who asserted the proposition? (Person or organization.) Should we believe the person or organization that asserted the proposition? In real-world cases (such as Web pages), typically we see complex assertions with implied dependencies on other Web resources: on commonly recognized knowledge (such as
Web Information Management
91
scientific laws); on sources not available on the Web (such as published papers or books); or on derived (or even anecdotal) knowledge. Relational links that define these dependencies then also require some form of provenance certification. No complete solution to knowledge provenance has been proposed as yet. However, a number of technologies in development address automated processing of assertions on at least the level of static provenance, building on older truth maintenance systems (TMS) and working towards working ontologies.
A Commercial or Free Web Although much of the public Web is free and open to anyone, and probably should remain that way for the common good, commercial interests must not be neglected. It is vital that as broad a spectrum of information as possible be made available, and some of that information is unavoidably connected with the business requirement to be a commercially viable offering; a return on investments. The debate about commercial contra free is often a heated one, and the common cry that ‘information wants to be free’ is often a misrepresented stance – and ultimately not true, as it is instead various people who want it to be free. Recent legislation changes to further lengthen copyright and restrict usage rights, and litigation and technology to enforce the new restrictions, are causing both furore and consternation among developers and in broad public awareness. Unfortunately, many of the restrictions and attempts to lock in perceived proprietary rights have already had or threaten to have a serious impact on the Web. For example, it can make it difficult or illegal (in some jurisdiction) to publish, re-use, or quote particular content. It can also make the development and deployment of sweb technology unexpectedly risk litigation due to claimed ‘foreseeability by a hypothetical reasonable person’ that users might infringe on content rights. Bit 3.20
Information might become dead before it becomes free
It is not just about free/freedom. Much ‘copyright management’ today has the deplorable result of making vast tracts of previously published information inaccessible, indefinitely, in any format or medium, at any price – not just for years, nor even for decades, but for entire lifetimes, plus 75 years (or the next copyright extension). It is not so much about such publicized issues as ‘piracy’ (a value-laden misnomer for intellectual property rights infringement), or ‘file sharing’ (and implication that p2p networking is inherently evil). These debated issues are admittedly legitimate concerns that suggest needed changes to how these rights are managed – or perhaps more properly, how issues of fair compensation to the creators and custodians of protected intellectual property be resolved. The real threats are more about outright grabs for basic and established Internet technology, such as the hyperlink mechanism, fundamental transport protocols, general architectural concepts, and established data formats or encoding types. The successful assertion of proprietary ownership of any of these could irreversibly cripple the general utility of the Internet.
The Semantic Web
92
Bit 3.21
Fundamentally free, for the greater public good
The W3C has taken the clear stance that insofar as implemented standards, protocols, and enabling technologies are concerned, the Internet as a whole must be free of patent claims – and remain free for the public good. This fundamental view apples equally to the emerging technologies of the Semantic Web. The landmark decision on W3C Patent Policy was formally taken in May 2003 (see summary at www.w3.org/2003/05/12director-patent-decision-public.html). Many participants in the original development of the Web could easily have sought patents on the work they contributed. They might subsequently have been tempted to secure exclusive access to these innovations or charge licensing fees for their use. However, they made the business decision that they and the entire world would benefit most from ubiquitous and royalty-free standards. CERN’s decision in 1993 (before the creation of the W3C) to provide unencumbered access to the basic Web protocols and software developed there, was critical to its rapid deployment and success. That the open platform of royalty-free standards enabled business to profit more from its use than from a Web based on proprietary ones is incontestable. Furthermore, the social benefits in the non-commercial realm are beyond simple economic valuation, in addition to laying the foundation for technical innovation, economic growth, and social advancement. The new policy commitment to royalty-free (RF) standards means
a stable, practical patent policy; a clear licensing framework; consistent disclosure obligations; an exception handling process when problems arise.
In practice, companies and individuals involved in the development of such components must therefore disclose and guarantee to waive any claims in order for these components to be accepted as part of the recommended specifications. Bit 3.22
The underlying Web infrastructure must be open and free
The Web is such an important part of modern society that the policy of public ‘greater good’ takes precedence over proprietary interests. Voluntary adherence to this principle is better than enforced compliance, since enforcement always has undesired side effects. Almost all companies, large or small, see the wisdom of this ‘greater good’ requirement and have signed on, which bodes well for the future of the Web. Otherwise, there is the established precedent that governments can go against business interests and by decree place critical patented technology in the public domain when proprietary interests threaten important public infrastructures. However, beyond the basic infrastructure, the Web platform, considerable scope for commercial ventures is possible – necessary, even. It all hinges on payment models.
Web Information Management
93
Payment Models E-commerce has grown rapidly in the existing Web, despite being hampered by various problems ranging from unrealistic business models to inappropriate pricing. In most cases, the relative successes have been the traditional brick-and-mortar businesses offering their products and services online, perhaps largely because they already had a viable business plan to build on, and an established trust relationship with a large customer base willing to take their shopping online. The advantage of real-world products and services in the Web context has been that transaction costs are drastically lowered, yet item prices are still comparable to offline offerings, albeit cheaper. It has, however, proven more difficult to construct a viable business model and pricing structure for digital content, in which category we include information ‘products’ such as music CDs, software and books, not just specifically Web-published content. Bit 3.23
Nobody really objects to paying for content – if the price is right
And, it must be added, if the mechanism for ordering and paying is ubiquitous (that is, globally valid), easy to use, and transaction secure. Arguably the greatest barrier to a successful business model for digital content has been the lack of a secure micropayment infrastructure. Existing pricing models for physical products are patently unrealistic in the face of the essentially zero cost to the user of copying and distributing e-content. Credit-card payment over SSL connections comes close to providing a ubiquitous payment model, but suffers from transaction costs that make smaller sums impractical and micropayment transactions impossible. Unfortunately, few of the large, global, content owners/distributors have realized this essential fact of online economics, and are thus still fighting a costly and messy losing battle against what they misleadingly call ‘piracy’ of digital content, without offering much in the way of realistic alternatives to sell the same content to customers in legal ways. Instead they threaten and sue ordinary people, who like most of their peers feel they have done no wrong. The debate is contentious, and the real losses have yet to be calculated, whether in terms of money or eroded public trust. It will take time to find a new and sustainable balance. Even fundamental, long-established policies of ‘fair use’ and academic citation, to name two, are being negated with little thought to the consequences. It can be questioned whose rights are being served by in practice indefinite copyright that renders huge volumes of literature, music, and art inaccessible to the public as a matter of principle. More disturbing is the insinuation by degree of ‘digital rights management’ (DRM) into consumer software and hardware, with serious consequences for legitimate use. DRMtechnology can time-limit or make impossible the viewing of any material not centrally registered and approved, even the blocking ability of creators to manage their own files. Perhaps the worst of this absurdity could have been avoided, if only one or more of the early-advocated technologies for digital online cash had materialized as a ubiquitous component integrated into the desktop-browser environment. For various and obscure
94
The Semantic Web
reasons, this event never happened, though not for lack of trying. Instead, online payment is currently based on: real-world credit cards (for real-world prices); subscription models (to aggregate small charges into sums viable for credit-card transactions); other real-world payment means; alternate payment channels that leverage these means (such as PayPal). The only relative success in the field has been the utilities-style rates model applied by ISPs for access and bandwidth charges, and some services with rates similar to phone usage. Bit 3.24
Free content can complement and promote traditional sales
A brave few have gone against commercial tradition by freely distributing base products – software, music, or books – figuring, correctly, that enough people will pay for added value (features or support) or physical copies (CD and manuals) to allow actually turning a profit. Baen Free Library (www.baen.com/library/) successfully applies the principle to published fiction. In the Semantic Web, an entirely new and potentially vast range of products and services arise, whose pricing (if set a price at all) must be ‘micro’ verging on the infinitesimal. Successful business models in this area must be even more innovative than what we have seen to date. One of the core issues connected with payment mechanisms is that of trust. The Semantic Web can implement workable yet user-transparent trust mechanisms in varying ways. Digital signatures and trust certificates are but the most visible components. Less developed are different ways to manage reputability and trust portability, as are schemes to integrate mainstream banking with Web transactions. Some technologies applicable to distributed services have been prototyped as proof-ofconcept systems using virtual economies in the p2p arena (see for example Mojo Nation, described in Chapter 8 of Peer to Peer), and derivative technologies based on them are gradually emerging in new forms as e-commerce or management solutions for enterprise. Bit 3.25
A Web economy might mainly be for resource management
It might be significant that the virtual economy prototypes turn out to be of greatest value as a way automatically to manage resources according to business rules. The Semantic Web might even give rise to a global virtual economy, significant in scope but not directly connected to real-world money. The distribution and brokerage of a virtual currency among users and services would serve the main purpose of implementing a workable system to manage Web resources. It is a known fact that entirely free resources tend to give rise to various forms of abuse – the so-called tragedy of the commons. Perhaps the most serious such abuse is
Web Information Management
95
the problem of spam in the free transport model of e-mail. It costs spam perpetrators little or nothing to send ten, or a thousand, or a million junk messages, so not surprisingly, it is millions that are sent by each. If each mail sent ‘cost’ the sender a small, yet non-zero sum, even only in a virtual currency tracking resource usage, most of these mails would not be sent. Various measures of price and value, set relative to user priority and availability of virtual funds, can thus enable automatic allocation of finite resources and realize load balancing.
4 Semantic Web Collaboration and Agency From information management in principle, we move on to the actors involved in realizing this management, mainly collaborating humans and automatic agents. The Semantic Web is slanted deliberately towards promoting and empowering both these actors by providing a base infrastructure with extensive support for relevant collaborative mechanisms. For most people, collaboration processes might seem to be relevant mainly to specialized academic and business goals – writing research papers, coordinating project teams, and so on. But it is about more than this narrow focus. Strictly speaking, most human communication efforts contain collaborative aspects to some degree, so that embedded mechanisms to facilitate collaboration are likely to be used in many more contexts than might be expected. Such added functionality is bound to transform the way we use the Web to accomplish any task. Planning and coordinating activities make up one major application area that involves almost everyone at some time or another. Current planning tools, such as paper agendas and electronic managers (PIM), started off as organizational aids specific to academic and business environments, yet they are more and more seen as essential personal accessories in daily life. They make daily life easier to manage by facilitating how we interact with others, and they remind us of our appointments and to-do items. With the spread of home computing and networks, PIMs are now often synchronized with both home and work systems. It is not hard to extrapolate the trend, factoring in ubiquitous Web access and the new collaboration technologies, to get a very interesting picture of a possible future where networked collaboration processes dominate most of what we do.
Chapter 4 at a Glance This chapter looks at a motivating application area of the Semantic Web, that of collaboration processes. Back to Basics reminds us that the Internet and the Web were conceived with collaborative purposes in mind.
The Semantic Web: Crafting Infrastructure for Agency Bo Leuf # 2006 John Wiley & Sons, Ltd
The Semantic Web
98
The Return of p2p explores the revitalization of p2p technologies applied to the Web and how these affect the implementation of the Semantic Web. WebDAV looks at embedding extended functionality for collaboration in the underlying Web protocol HTTP. Peer Aspects discusses issues of distributed storage, security and governance. Peering the Data explores the consequences and implications of a distributed model for data storage on the Web. Peering the Services looks at Web services implemented as collaborating distributed resources. Edge Computing examines how Web functionality moves into smaller and more portable devices, with a focus on pervasive connectivity and ubiquitous computing. Automation and Agency discusses collaboration mechanisms that are to be endpointagnostic, in that it will not matter if the actor is human or machine. Kinds of Agents defines the concept and provides an overview of agent research areas. Multi-Agent Systems explores a field highly relevant to Semantic Web implementations, that of independent yet cooperating agents.
Back to Basics For a long time (in the Web way of reckoning) information flow has been dominated by server-client ‘broadcasting’ from an actively-publishing few to the passively-reading many. However, peer collaboration on the Web is once again coming to the fore, this time in new forms. As some would say: It’s about time! The original write-openness of the Web was largely obscured in the latter half of the 1990s, as millions surfed into the ever-expanding universe of read-only pages. Bit 4.1
Not everyone sees themselves as Web content authors
The objection that most people are perfectly content to just read the work of others does, however, beg the question of how you define ‘published content’. Much useful information other than written text can be ‘published’ and made accessible to others. There were, of course, good reasons why the Web’s inherent capability for anyone to ‘put’ new content on any server was disabled, but a number of interesting potential features of the Web were lost in the process. Reviving some of them, in controlled and new ways, is one of the aims of the Semantic Web initiative. Content-collaboration became needlessly difficult when the Web was locked down. Publishing on the Web required not just access to a hosting server, but also different applications to create, upload and view a Web page. Free hosting made way for commercial hosting. The large content producers came to dominate, and portal sites tried to lock in ever more users in order to attract advertising revenue. In fact, for a brief period in the late 1990s,
Semantic Web Collaboration and Agency
99
it almost appeared as if the Web was turning into a just another broadcast medium for the media corporations. Everyone was pushing ‘push’ technology, where users were expected to subscribe to various ‘channels’ of predigested information that would automatically end up on the user’s desktop. Collaboration became marginalized, relegated to the context of proprietary corporate software suites, and adapted to conferencing or ‘net-meeting’ connections that depended on special clients, special servers, and closed protocols. Or so it seemed. Actually, free and open collaboration was alive and well, only out of the spotlight and awareness of the majority of users. It was alive between researchers and between students at the institutions and universities that had originally defined the Internet. It was also alive in a growing community of open-source development, with a main focus on Linux, even as the grip of commercial interests around the PC market and by extension the Web was hardening. The concepts were also alive among the many architects of the Internet, and being refined for introduction at a new level. Developers were using the prototypes to test deployment viability. Finally, there was the vision of a new Internet, a next-generation Web that would not just provide a cornucopia of content but one of functionality as well. The line between ‘my local information’ and ‘out-there Web information’ seems to blur in this perspective, because people tend both to want access to their own private data from anywhere, and to share selected portions of it for use by others (for example, calendar data to plan events). The server-centric role of old then loses dominance and relevance. That is not to say that server publishing by sole content creators is dead – far from it. Ever increasing numbers of people are involved in publishing their own Web sites or writing Web logs (blogs), instead of merely surfing the sites of others. Such activity may never encompass the masses – the condescending comment is usually along the lines that most people simply have nothing to say, and therefore (implied ‘we’) do not need the tools. But that is not the point. Bit 4.2
The capability of potential use enables actual use when needed
Providing a good education for the masses does not mean that we expect everyone to become rocket scientists. However, we can never know in advance who will need that education, or when, in order to make critical advances for society as a whole. We do not know in advance who will have something important to say. Therefore, the best we can do is to provide the means, so that when somebody does have something to say, they can do it effectively. Anyway, with increased interaction between user systems, distributed functionality and better tools, more users will in fact have more to say – possibly delegated to their agent software that will say it for them, as background negotiations with the remote services.
The Return of p2p In the late 1990s, peer-to-peer technology suddenly caught everyone’s fancy, mainly in the form of Internet music swapping. It was a fortuitous joining of several independent developments that complemented each other:
The Semantic Web
100
The explosion in the number of casual users with Internet access – everyone could play. The development of lossy yet perceptionally acceptable compression technology for nearCD-quality music – leading to file sizes even modem users could manage (4 MB instead of 40). The new offering of small clients that could search for and retrieve files from other connected PCs – literally millions of music tracks at one’s fingertips. Legalities about digital content rights (or control) aside (an area that years later is still highly contentious and infected in the various camps), here was a ‘killer application’ for the Internet that rivalled e-mail – it was something everyone wanted to do online. But the surge in p2p interest fostered more than file-swapping. Bit 4.3
Peer-to-peer is fundamentally an architectural attitude
System architecture always profoundly reflects the mindset of the designers. In this case, p2p and client-server are opposing yet complementary attitudes that determine much more than just implementation detail. New and varied virtual networks between peer clients were devised, developing and testing components that prove critical to many Semantic Web application areas, including collaboration. The reason they prove critical is that p2p networks faced many of the same design problems, albeit on a smaller scale. These problems are often related to issues of identity and resource management in distributed networks. With client-server architectures, most of these issues are resolved through centralized registry and governance mechanisms depending on some single authority. Assuredly, there are situations when this rigid and controlled approach is appropriate, but the approach is not conducive to ease of use, nor to the establishing of informal and fluid collaborations over the Web. Peer-managed Mechanisms In implementation, we can envision two peer-like classes of entities evolving to meet the new paradigm of distributed functionality: Controller-viewer devices, sort of like a PDA with a screen and a real or projected keyboard, that are the user’s usual proximate interface and local storage device. From it, control and views can be handed off to larger systems, such as a notebook, desktop, or room-integrated system with wall-projection screens. You might see it as a remote control on steroids. Agent-actuator devices, both physical and software, which negotiate data and interact with the user on demand, and function as the intermediaries between the user-local PDA views and the world at large. Reviewer Austin David suggested, in the course of the first review of this book’s text, the following vision of what this situation might be like for an average user:
Semantic Web Collaboration and Agency
101
My various views would interact with my personal agent, and I would carry the views on my person, or leave them attached to my car or bedroom wall, or in my toaster or radio. The agent would know my calendar, or how I like my toast, or what CD I want to hear in the morning, and would be ‘out there’ to interact with the rest of the Semantic Web.
Since its foundation, the Web has been eminently decentralized and based on consensus cooperation among its participants. Its standards are recommendations, its ruling bodies advisory, and its overall compliance driven by individual self-interest rather than absolute regulation. Collaboration processes share these characteristics. Not surprising, since the ongoing development of the Internet as a whole rests on collaborative efforts. Why, therefore, should small-scale collaborative efforts be locked into tools and protocols that reflect a different, server-centric mindset? New Forms of Collaboration Many Web resources are repositories for information gathered from diverse, geographically separated sources. The traditional, centrally managed server model has the bottleneck that all updates must pass through one or more ‘webmasters’ to be published and become accessible to others. Bit 4.4
In the new Web, sources are responsible for their resources
Although implicit in the original Web, this distributed responsibility became largely replaced by the webmaster-funnel model of information updating on servers. When primary sources can author and update information directly to the server, without intermediaries, great benefits result. A small-scale, informal version of such co-authoring was explored in The Wiki Way, a technology that used existing Web and client-server technology but in an innovative way. The new Web seeks to implement comparable features pervasively, as basic functionality built into the infrastructure’s defining protocols.
WebDAV Web Distributed Authoring and Versioning, WebDAV (see www.webdav.org), or often just DAV, is a proposed protocol extension to HTTP/1.1. It has for a time been gathering significant support in the Web community. DAV adds new methods and headers for greater base functionality, while still remaining backwards compatible with existing HTTP. As the full name implies, much of the new functionality has to do with support for distributed authoring and content management – in effect, promoting collaboration at the level of the base Web protocol. In addition, DAV specifies how to use the new extensions, how to format request and response bodies, how existing HTTP behavior may change, and so on. Unlike many API and overlay protocols that have been developed for specific collaboration purposes, DAV intends
The Semantic Web
102
to implement a generic infrastructure support for such functionality, thereby simplifying application development. Bit 4.5
DAV is a basic transport protocol, not in itself semantic capable
The point of DAV in the context of the Semantic Web, however, is that it enables transparently much functionality that makes it easier to implement semantic features. In the context of co-authoring, DAV is media-agnostic – the protocol supports the authoring (and versioning) of any content media type, not just HTML or text as some people assume. Thus, it can support any application data format. The Basic Features DAV provides a network protocol for creating interoperable, collaborative applications. Major features of the protocol include: Locking, which implements well-defined concurrency control to manage situations where two or more collaborators write to the same resource without first merging changes. Properties, based on XML, which provide storage for arbitrary metadata, such as a list of authors on Web resources. Namespace manipulation, which support copy and move operations for resources. DAV implements DASL (DAV Searching and Locating protocol), to perform searches based on property values to locate Web resources. DAV Extensions Several extensions to the base DAV protocol are developed in the IETF: Advanced Collections, which adds support for ordered collections and referential resources (or pointers). Versioning and Configuration Mangement, which adds versioning support, similar to that provided by RCS or SCCS. The configuration management layer, based on Versioning, provides support for workspaces and configurations. Access Control, which is implemented as the ability to set and clear access control lists. In a generalized form, access control can be applied to the entire Web. Access control will always be a factor, no matter how ‘free’ and ‘open’ the Web is styled. The difference in the open context is that access and governance methods are decentralized and firmly based on semantic models of identity/authority assertion. Strong public-key cryptography, with its associated digital signatures, is the enabling technology that makes such methods feasible. These issues are discussed in Chapter 3. The future trust model is one of overlapping ‘webs of trust’ that complement existing central trust authorities; an interoperative system that can adequately establish secure identities on demand.
Semantic Web Collaboration and Agency
103
Peer Aspects In the Semantic Web model, data and metadata are often moved away from central stores and into distributed storage all over the Web. In many cases, the primary user focus also shifts from the data as such to packaging services that deliver the data preprocessed or aggregated in various ways. Data governance then takes on a different cast, because the usual physical locality of data no longer applies, and the traditional security mechanisms become inadequate or inappropriate. Bit 4.6
New consistent endpoint-to-endpoint security models are needed
A new Semantic Web infrastructure to handle security and integrity issues consistently is in fact being developed, leveraging existing endpoint encryption technologies. The section on XML Security in Chapter 3 provides an overview of the new security infrastructure. Many of the issues of distributed storage, security and governance may be explored from the perspective of existing p2p systems that are built on top of the current Web. Much of this material is still valid and applicable to many aspects of sweb architecture. Despite the recent wave of hostility towards p2p systems, generally due to intense litigation and threats by representatives of the music industry against both hosts and developers, p2p as a technology (or rather a collection of technologies) is very much a part of the mainstream development of the Web towards the Semantic Web. The following sections revisit and expand on some of these p2p themes.
Peering the Data Distributed (and peer-managed) data have many advantages over centralized stores. However, the actuality is not an either/or situation, because server sourcing and distributed sourcing complement each other and can create a valuable synergy that leverages the best of both models. Most content-creating users are for the foreseeable future still likely to publish to specified server locations. In only a few special situations is content published to (or ‘inserted’ into) the virtual network without any specified location. Instead, it is the retrieval and subsequent processing of published data that will occur in a distributed and non-localized manner. In most cases, the user is not very interested in where the data reside, only that they can be found quickly and reliably, and presented in meaningful ways. Bit 4.7
Usage patterns show that answers to ‘what’ are preferred to ‘where’
The popularity of content-aggregating services (such as portal sites) suggests that users often prefer the convenience of subscribing to dereferenced compilations of desired content from a single source, instead of having to browse around to visit all the individual source sites.
104
The Semantic Web
Given a decent rules-following client, the same users will probably prefer to delegate ‘find-and-fetch’ tasks to the automations. In such a model, it is an advantage to have content addressed by identity rather than tied to locations. URI addressing in turn makes storage at consistent locations a non-issue, entirely optional. Swarming The synergy of server-peer is clearly evident in ‘swarming’ technologies built around sourcing data from a single publishing source. The collective distribution power of all the interested clients is used to reduce loading on the server by passing along data fragments already received to others requesting them. Practical demonstrations of the power of swarming technologies can be a dramatic experience for anyone with a ‘fat pipe’ connection (for example, multi-megabit/s ADSL). A client request is sent to some mediating server, which results in a download of the requested data starting. The transfer trundles along at a paltry few kilobyte/s for a while, but then the rate rapidly increases as more swarming clients are contacted in the background. Very soon, the download bandwidth is saturated, despite modest requirements on the individual sourcing nodes. Asymmetric connectivity (as in prevailing ADSL) provides significantly lower upload than download bandwidth, so swarming ensures maximum transfer rates to the downloading user. At the same time, as more fragments of the file are received, the receiving client begins to source them to other members of the swarm who lack just those fragments. Swarming techniques may be more important in the context of adaptive load distribution in general, rather than just maximizing usage of individual connection bandwidth. A typical side-effect is that requested data are replicated and move closer to where the most demand is located. Subsequent requests from ‘nearby’ clients therefore need fewer node hops to access the same data. This caching is beneficial to the network as a whole. Combining data fragmentation, encryption, and redundant replication of the chunks gains security and a measure of independence from individual servers. Clients instead rely on distributed services to locate and retrieve the data on demand from whatever stores are available at any given time. Identity, not Location A major design issue when distributing data and metadata over the network, especially in adaptive networks, is the desire to decouple access from a particular storage location and exposed access method on an identified server – the hitherto pervasive URL-based identity model. Therefore, the Semantic Web instead presumes that the location-agnostic URI is the normal access identity for resources – and for that matter for the individual. Distributed retrieval services then handle the messy details of search and retrieval, dereferencing the URI to the specific Internet IP-location. Such a URI can have many different forms, including the totally opaque digital-key, content-hash model used in Freenet (freenet.sourceforge.net). In fact, nowhere is the
Semantic Web Collaboration and Agency
105
decoupling of data from locality more evident than in this model, where physical data location is smeared out though distributed caching and anonymization services to the extent that it is ‘nowhere and everywhere’ all at once. A Freenet-like architecture, however, is an extreme solution for achieving very specific goals, including: guaranteed anonymity of both publisher and reader; consistently verifiable identity despite anonymity; guaranteed immunity to data change and censorship attempts. As such, it is not likely to be the norm.
Storage Model Nevertheless, we can expect a much more virtual storage model to proliferate. The user will still see ‘discrete documents’ in some kind of information space, and the metaphor of server storage in a hierarchical filesystem may remain a viable one, but the physical reality might look very different. Unused storage space on various physical networks, such as corporate networks, might become a freely traded commodity. Securely encrypted fragments of public and private document data may then be redundantly spread across arbitrarily chosen rented disk capacity. Locator services would transparently handle storage and retrieval through distributed broker services. The model is really not that strange – the local filesystem on a PC or a server already performs a similar dereferencing service, invisibly translating through several layers between the visual user model of a hierarchical system of directories and files displayed in a window, and the physical locator model of heads, cylinders, and sectors on the disk. The big difference is one of scale, of effectively turning the entire Web into an extended filesystem with an additional layer of descriptive metadata. The future user of the Web will neither know nor especially care where the data are stored physically, except perhaps in special circumstances, as long as it is secure and reliably accessible. The immediate benefit is that the user can access data from anywhere on the Web; control of the data, instead of being is rather than being locked up in one or more specific machines, is mediated by some authority model based on personal identity.
Peering the Services In addition to distributing data across the network, also distributing and delocalizing functionality reaps great benefits. This further step is an integral part of the Semantic Web model, because distributed data one just a special case of generic URI-addressed resources that also include functionality and services. Implementing distributed services, however, does pose greater demands on the ability of the components to communicate and coordinate with each other when fulfilling requested tasks. The Semantic Web aims to provide the infrastructure and mechanisms to make such coordination easier.
The Semantic Web
106
Although many have written much about ‘Web Services’ in the past few years, not everyone means the same thing by the term. Nor do all implementations have very much to do with the Semantic Web model of distributed services (SWS, see discussion of WS in Chapter 3). WS products are sometimes designed as traditional client-server architecture – centralized, monolithic, and controlled from a single location on the Web. While some services might be appropriate to implement in this way, most are probably not. The choice of implementation typically depends on several factors: the type of service considered, QoS required, and the risks at stake. Bit 4.8 Web Services is a very broad term that covers many different kinds of implementations of greatly varying complexity A Web service involved in medical applications (an early-adoption area for SWS) requires different constraints than a similar service for consumer product delivery. Significant services are more likely to be implemented as highly distributed ‘swarms’ of adaptive components that can self-organize to meet the demands of the moment. It is more than likely that many of these will be realized in the form of connected devices near the users, while others might take the form of software manifesting virtual entities with ill-defined locality on the Web.
Edge Computing As it happens, ‘edge computing’ devices are a central feature of the Semantic Web’s focus on pervasive connectivity and ubiquitous computing. On the one side, the trend is towards smaller and more portable, yet ever more capable hand-held devices, starting to replace notebooks as they in turn replace desktops. Converging from the other side is the trend to add more computing power and wireless connectivity to other previously ‘dumb’ peripheral devices, until practically everything is programmable. The sheer number and variety of such devices, and the many arbitrary combinations that might arise in a cooperative effort between them, precludes any way of effectively defining all situation parameters in advance. Therefore, the workable approach is to empower devices to discover each other independently and negotiate their own terms for cooperation. Bit 4.9
Automatic connectivity, discovery, and negotiation will characterize devices
Embedding silicon and software is cheap. Typically, mass-produced devices appear with ever more ‘free’ functionality each year. Ubiquitous remote connectivity makes a transforming level of available features attractive to manufacturers. The Semantic Web model addresses this functionality only indirectly, by defining the protocols and languages involved to implement it, and the overall framework for cooperative computing. See the following section on agents.
Semantic Web Collaboration and Agency
107
Smart and connected devices have not yet caught on in the market place. PDA/PIMdevices with wireless connectivity and the most recent 3G cellular phones are but precursors to the possibilities. As yet these devices suffer the problem that the more interesting potential services and nodes are just not there yet, or that connectivity is tied to proprietary solutions. Early adopters are used to paying premium prices for first-on-the-block gadgets, yet some of the offered prototypes of Web-aware appliances have not caught even their fancy. Web-aware refrigerators, for example, have been around for years in one form or another, at ten times the price of normal ones, but clearly lack appeal in the absence of a complete home network to integrate more than just milk monitoring. I am reminded of early PC marketing touting large-footprint desktop machines for the home as just the thing to organize dinner recipes. Yet eventually a critical threshold was passed and several PCs in every home became a common situation. Edge computing is an expression of ‘little engineering’, which concerns the common products that end up being produced in great volume. These products then cost comparatively little for each unit, and ultimately they redefine everyday life. Examples are legion: cars, household appliances, cameras, PCs, PIMs, and cellular phones – yes, especially the cellular, with units practically given away on promise of operator tie-in subscription. Bit 4.10
‘Little engineering’ inherently drives significant social change
It does this not by solving any great problems (which is the purpose of ‘big engineering’), but by driving down the cost of a known technology solution so that everyone (or at least the vast majority) can afford to use it. Little engineering also critically depends on the existence of a standardized and available infrastructure, usually quite complex: cars need roads and filling stations and refineries, appliances need wall socket wiring and electric grid production utilities, cellular phones need operators and vast reaches of repeater masts every few miles, and so on. Without the infrastructure, the gadgets are useless – consider how much of what we use on a daily basis would not work on a remote desert island. Therefore, the watershed event in edge-computing will not be any single, whiz-bang, Webaware gadget (though assuredly over the years there will be a succession of hopeful candidates). No, in this context, it is simply the cumulative effect of a convergence of two developments: Ever more gadgets sold in new versions with always-on connectivity and agent software built in, ready to configure and use. An expansion of always-on or on-demand network connectivity to cover work and home environments, available as a normal infrastructure the same way as electricity. Assuming the infrastructure exists for the devices and software, let us examine what may drive the device functionality.
The Semantic Web
108
Automation and Agency Deploying generic mechanisms on the Web to facilitate collaboration between humans is important in its own right, but the Semantic Web advocates a much broader view. Collaboration mechanisms are to be endpoint-agnostic – in other words, to facilitate the full span of possible collaboration processes: human and human (mediated by the tools and protocols used, often called PP in the sense of person-to-person, or even people-to-people, as a variant of the technical term p2p for peer to peer); human and machine (person to software agent, often called PA); machine and machine (agent to agent, often called AA). Overall, collaboration is a more abstract and broader process than just interoperability – the latter concerns the fundamental technical ability to communicate, basically a matter of protocol and specifications, whereas collaboration processes are more concerned with the semantics of what is being communicated. Bit 4.11
Collaborative processes imply meaningful conversations
Collaboration requires an exchange of information, of messages understood. Idiomatically speaking, all three collaboration processes (PP, PA, and AA) should be seen as conversations imbued with meaning. What the Semantic Web brings to the situation are the predicated-based structures to express meaningful assertions, and the ontology back-end to parse meaning from these sentences. That is not to say agents will be able to ‘think’ like their human counterparts, but rather that they will be able to reason logically around relationships (explicit or implicit) and infer new valid ones. Semantic reasoning capability is especially valuable in any conversations of type PA, because humans lapse easily into implicit referencing. Even such a ‘simple’ PA command as ‘print this document on the printer to my left’ would require considerable contextual inference by a software agent, regardless of whether it could interpret any supporting ‘gestures’ by the user. Bit 4.12
Human semantic messages are complex and multi-modal
It is very difficult to get machines to understand lexically even artificially constrained human idiomatic usage. Conversely, humans are frequently frustrated by the constraints of artificial command languages and syntax. Semantic processing in machines can bridge the gap and hide the complexities. Further supportive mechanisms include automatic identification and verification of authority (including delegation), automatic resource discovery (local devices, their capabilities, and availability), and appropriate levels of security (encryption). Granted that much of this process support is in some sense already available in the traditional Web, but it is then
Semantic Web Collaboration and Agency
109
embedded in particular tools, special protocols, and special manual procedures. Invariably, it is an exception, an add-on, rarely convenient to use. The point of the Semantic Web is to make such support pervasive and transparent, a part of the fundamental protocols of the Web, always available from anywhere.
Kinds of Agents ‘Agent’ may need further expansion here. It is a term increasingly used (and at times misused), often without any special qualification, to describe a wide range of disparate computational entities. A selection of ‘classic’ definitions of agent include: a persistent software entity dedicated to a specific purpose; a program that emulates a human relationship by doing what another person could do for the user; an integrated reasoning process. An e-business quip described an agent as ‘a credit card with an attitude’. The decision to call any entity an agent ultimately rests on our expectations and point of view. Sometimes it is just convenient for both developers and users to see a particular program as a ‘rational agent’ based on an interpretation of its behavior, rather than as an ordinary piece of software. Then, it becomes natural to talk in terms of beliefs, desires, intentions, and goals when designing and analyzing the behavior. In AI literature, this particular anthropomorphic attitude is sometimes referred to as ‘Dennett’s intentional stance’, as distinct from the earlier two predictive stances of physical object and functional design. Bit 4.13
‘Agent’ is a descriptive term of convenience, not a well-defined one
Although there is some agreement on general characteristics of ‘intelligent’ agents, the decision to call a piece of software an agent is an arbitrary one. Otherwise, a more formal description of agent capabilities tends to highlight many features that distinguish agents from ordinary application software: autonomous behavior, meaning they can act independently on behalf of the user to perform delegated, goal-oriented tasks; reactive behavior, meaning that they can perceive and respond to changes in their environment; proactive behavior, or goal-directed initiative, so they do more than just react to events or input; social ability, so they can communicate with the user, the system, and other agents, as required; intelligence, meaning reasoning at some level, from fixed rules to heuristic engines, that allows them to adapt to changes in their environment;
The Semantic Web
110
cooperation, so they can work with other agents to perform more complex tasks than they themselves can handle; mobility, so that they may move between host systems to access remote resources or arrange local interactions with remote agents. A given agent implementation need not exhibit all of these features; it depends as noted on the expectations and point of view. Simple cooperative agents might for example be entirely reactive. A number of different graphical maps promoted in the literature attempt to place agents into more systematic groups, for example with schemes that overlap in nice geometric ways to indicate agent purpose areas. Agent use cases are generally classified in two groups: intelligent resource managers that simplify distributed computing; personal assistants that adapt to user requirements. Both of these areas are especially relevant to telecommunication companies, who not surprisingly also account for the largest portion of agent research. Evidently, they foresee enough of a return on the investment to continue. A common approach is that agents provide intelligent interfaces that ‘wrap’ around legacy systems and provide new features to established functionality. Often the enhanced functionality comes from exposing previously local controls to remote access. Examples include distributed network management and self-help customer care systems. Such leveraging of existing technology seems a viable development path for much of the transition to the Semantic Web as well. Bit 4.14
Agent technology implements better interfaces to existing resources
We already see powerful trends to expose Web-resource APIs to client-side programming (such as in a browser toolbar or applet). Such access can transform simple site functionality into more complex services that are called from user instances of customized interfaces. Conceptually, agents have a long history, even in the context of the Web. As the name suggests, DARPA Agent Markup Language (DAML, see the DARPA section of Chapter 7) was considered as a special markup solution for agents. This approach changed as several strands of development joined to form more powerful descriptive structures incorporating meaning relations, and later the basis for OWL (described in Chapter 6). As yet, however, there are few indications of any dramatic ‘killer application’ to catapult agent technologies into the mainstream of user awareness – or for that matter, provider awareness. Instead, we might be tempted to call them ‘secret agents’ because many are slowly diffusing out into undercover deployment. Garbed in traditional Web form interfaces, agents perform background AA tasks such as cooperative search without the immediate user realizing it. As more small-device processing becomes the norm, such as users wanting to access WS from small hand-held devices, agents appear highly attractive. Mobile agents can take tasks
Semantic Web Collaboration and Agency
111
into the network and to other hosts for processing, tasks that would overwhelm the capacity of the original hand-held host. Later they would return with results for display on the handheld, or automatically forward more complex displays to more capable systems in proximity to the user. Bit 4.15
Portable computing is likely to spawn many agent applications
This application area may be where we can expect the first killer applications to appear. Agents Interacting Agents are components in an agent infrastructure on the Web (or Web-like networks) – in the first analysis, interacting with services and each other. Such an agent infrastructure requires a kind of ‘information food chain’ where every part of the food chain provides information that enables the existence of the next part. The chain environment is described by the ontology (see Chapter 6) of the domain. Each agent’s own ‘understanding’ is not meant to describe all aspects of the domain, however, as it is sufficient to scope it to the agent’s context and view. Views necessary for particular tasks may vary between different agents, perhaps designed around special ontologies, so the overall system design must handle and reconcile multiple viewpoints. The problem-solving behavior of agents requires some kind of task decomposition approach, preferably as reusable components. The chosen approach, new to software agents, is essentially imported from task-method dichotomy from the field of Knowledge Engineering, where a task is decomposed into corresponding subtasks. This decomposition structure is iteratively refined down to a level at which primitive methods (so-called mechanisms) can solve the subtasks. Interoperability between agents requires interoperability between ontologies, which is usually solved by mapping strategies between them. Various wrapper agents also need to be devised to transform data representations from locally stored structures at the information sources to the structures represented in the ontologies. The Ontobroker project (see ontobroker.semanticweb.org for a W3C overview) was an early attempt to annotate and wrap Web documents and thus provide a generic answering service for individual agents. Figure 4.1 illustrates the general architecture of query agents in this service. However, an infrastructure of interacting agents is increasingly seen as better implemented as a community of cooperating agents. This shift requires a different model of behaviour than simple query agents around information brokers. More fluid and adaptable responses are expected, and individual agents must be better equipped to appraise independently their situation in relation to specified goals.
Multi-Agent Systems In the AA arena, we should look to current research in Multi-Agent Systems (MAS). The focus in MAS research is on systems in which many ‘intelligent’ agents (meaning autonomous entities with significant semantic logic capability) interact with each other to perform tasks. Such entities can be software programs or robots (devices).
The Semantic Web
112
Figure 4.1 A schematic over Ontobroker architecture. The service functions as an automatic answering service for queries coming from external clients or agents, retrieving preprocessed, wrapped metadata
These systems are characterized by a number of basic tenants: Each agent on its own has incomplete information or capabilities for solving the assigned task – that is, a limited viewpoint. There is no system-wide, global control of the entities. Data sources are decentralized, and typically must be discovered. Computation is asynchronous, and communication is message-based. Agent interactions may further be characterized as either cooperative, sharing a common goal, or selfish, pursuing agent-centric goals. Bit 4.16
Multi-agent systems are comprised of autonomous agents
They may be seen as task-oriented swarms of activity, automatically forming and dissolving according to overall requirements and agent-goal fulfilment. Figure 4.2 indicates the relationships from the MAS perspective.
Figure 4.2 MAS architecture consists of independent agents interacting and collaborating with each other around tasks, and often implementing services for other agents
Semantic Web Collaboration and Agency
113
The research areas of multi-agent systems and distributed systems coincide, hence ‘distributed agent computing’ as a research area on its own. Multi-agent systems are often distributed systems, and distributed systems are platforms to support multi-agent systems. MAS researchers develop communications languages, interaction protocols, and agent architectures that facilitate this development. They can leverage sweb work done on descriptive markup, message protocols, and ontologies. Bit 4.17
Multi-agent design is often inspired by real-world interaction models
MAS research draws on ideas from many disciplines outside of AI, including biology, sociology, economics, organization and management science, complex systems analysis, and philosophy. Interaction analogies are often made with models derived from corresponding interactions seen in nature or in human society – such as swarms, ant colonies, or the free economic market. A common experience from prototypes based on such designs is that surprisingly simple, agent-centric (that is, selfish) rules can generate complex behavior that benefits the collaborative goals. For example, a MAS researcher may design rules so that a group of selfish agents will work together to accomplish a given task, basing the work on studies of how individual ants collectively gather food for the ant hill. A useful portal page to find important online publications and overviews on the various research areas of agents, MAS, and related fields is maintained at the AAAI (www.aaai.org/AITopics/html/multi.html). More news-oriented links are posted at MultiAgent Systems (www.multiagent.com). Information Agents A special category of agents is identified with the purpose of managing the explosive growth of information. They are designed to manipulate or collate information from many distributed sources, and early implementations have been around on the Web without users even reflecting on the agent technology that powers the functionality. Bit 4.18
Information agents often function as intermediary interfaces
In this capacity, they remove the requirement that the user finds and learns the specific interfaces for the actual information resources. For example, meta-search engines parcel out user queries to many different search engines, process the many result lists, and return compiled and filtered results to the user. Web price-comparison services collect information from various online shops to present comparison tables for selected products. Other implementations search out and index published research papers, providing citation searching and other academic services.
The Semantic Web
114
An overview of the seminal research projects that led to some of the services of this nature that are currently deployed on the Web may be found at the Internet Softbot Research site (www.cs.Washington.edu/research/projects/WebWarel/www/softbots/projects.html ). Economic Agents A particular model of cooperative agents centers on interactions governed by business logic in a virtual economy. Agents ‘buy’ and ‘sell’ resources to manage information in a kind of free market, with the overall goal of accomplishing set tasks. Distributed swarms of agents can act in different roles depending on ‘market pressure’ and internal rules (for example, provide more instances of a scarce resource by more agents responding to a higher offered ‘bid’ for the service), or optimize critical performance (by bidding higher to contract premium services or bypass queues). Bit 4.19
Market-based agent behavior can optimize resource usage/cost
Importantly, market-based solutions can virtualize and commoditize even the user’s idle resources, and through background trading on the common agent market provide a transparent way to offset costs due to their own consumption of network resources. The main advantages of such market-oriented programming are minimal communications for effective cooperation, automatic load balancing, and adaptive response to new requirements. Economic science has developed an extensive analytic framework for studying such systems. The model is naturally well-adapted to management applications for physical network, but it has also been used extensively to build p2p virtual networks to provide adaptive storage and processing. The latter are in effect precursors to collaborative Web services.
Part II Current Technology Overview
5 Languages and Protocols Moving along from the general discussions about concepts and function, we need to look at how to express it all, and how to structure the data and metadata so that they are usable. Structure is an old concept, though most often it is something we are so used to that we take it for granted. Language and grammar, nuance and styling, layout and conventions – all make a ‘texture’ against which we paint our semantic meanings. Humans have an innate ability to analyze and structure sensory data, discovering and sometimes inventing patterns to order seeming chaos, and inventing conventions to impose corresponding patterns on recorded data. The trouble is, most human patterns are not well-suited to the current requirements of machine-parsed patterns. Computer programming languages have therefore always come across as cryptic and non-intuitive – mainly because machines lack that ability we call intuition and must for this reason be directed in excruciating detail, step by step. Ongoing research has tried to address this deficiency by devising logic systems that exhibit something closer to the human mode of reasoning and that can reason and infer in limited ways from data and rulesets. In the Semantic Web, it is no longer a question of isolated machines and their local processing or reasoning. Instead, a common approach is required so that the local software can not only reason about the data and metadata but also actively collaborate with other bits of software elsewhere. Languages and protocols must enable a free and implementationindependent exchange of structure based on some sort of common ‘understanding’ about what the data mean. For this reason, part II is an exploration of the information technologies that define and enable structure and reasoning. We start with the basic building blocks and work up to more advanced structures – an overview but with some history.
Chapter 5 at a Glance Chapter 5 is concerned mainly with the building blocks that structure the data, and enable the definition and expression of metadata assertions. Closely related to the structure are the protocols that define the process of managing the structures. Markup gives an overview of the fundamental methods for tagging structures and meta-information on the Web. The Semantic Web: Crafting Infrastructure for Agency Bo Leuf # 2006 John Wiley & Sons, Ltd
118
The Semantic Web
The Bare Bones (HTML) looks at the cornerstone of the published Web as we know it, the hypertext tag. Disentangling the Logic (XHTML and XML) explores the evolution of simple Web markup, from just tagging text to make it ‘look good’ into something that better supports codifying meaning. Emerging Special-Purpose Markup takes note of some of the newer, lightweight markup standards that bring XML to small devices. Higher-Level Protocols takes up the message-based protocols that leverage XML to create more object-oriented and semantic-based exchanges. A Messaging Framework (SOAP versus REST) shows examples of how XML-based messages can define a common framework for packaging and exchanging XML-tagged content and functionality. It also critiques SOAP application by referring to the REST approach. Expressing Assertions (RDF) explains the logic and syntax behind the metadata structures (usually expressed in XML) that can define relationships between data elements and reference other resources to help define meaning for subsequent processing. The Graphical Approach introduces an alternative or complement to RDF, intended to deal with rich structures difficult to describe in a linear fashion. Exchanging Metadata (XMI) discusses an enterprise-level standard for exchanging metadata in distributed networks. Applying Semantic Rules explores proposed standards for identifying and applying knowledge-base rulesets to knowledge-management databases (‘expert systems’). RDF Query Languages examines the process of extracting metadata from the RDF structures that define and express the relationships. Finally, Multi-Agent Protocols describes an emergent area of defining agent interaction using protocols capable of semantic expression.
Markup And the Tag did impart structure upon the face of the Web. . . . Genesis, the Web version Markup languages represent the technology for encoding metadata into the content itself. Markup tags are the specific codes or patterns that ‘escape’ the literal interpretation and rendering for the duration of the metadata blocks. Simple punctuation, as found in this text, is just another form of markup. More subtle forms are styling and layout in printed works. Electronic storage allowed much more intricate and well-defined markup solutions. Bit 5.1
Markup is any arbitrary convention for embedded metadata
What ‘embedding’ means can vary greatly depending on the medium and the kind of ‘processing’ available to the end-user. Processing rules can be implicit (shared conventions), explicit (described literally as text), or self-defining (by reference).
Languages and Protocols
119
Advanced markup languages were in fact devised long before the Web. The case can be made that HTML, a very simplified subset of SGML, is what really defined and empowered the Web. This markup gave a consistent structure to published content, and above all, it defined the powerful hyperlink mechanism that could express a simple form of relationships between documents. So although many of the fundamental components of the Web were inherited from earlier application domains, including the hyperlink concept from Hyperstack cards and electronic texts, the synergy of combining them all with simple protocols for large-scale, open ‘opt-in’ networks proved unimaginably powerful.
The Bare Bones (HTML) Although HTML was in principle a structural markup system, it quickly acquired a number of tags and usage conventions that targeted primarily visual presentation for human readers of Web content. This was an unfortunate, but perhaps inevitable, development in the circumstances. It was only partially rectified with the later introduction of a stylesheet layer (CSS) to separate visual layout attributes from the actual content markup. A second problem with HTML is that the tag set was arbitrarily fixed according to an external standard. Over time this standard was extended in various ways for different reasons by different parties. Client development targeted mainly new visual presentation features using the new tags, with little regard for either usability or meaningful markup. In the HTML 3.x versions, for example, we saw these extensions become a part of the ‘browser wars’ as different vendors strove to dominate the Web. The deployment of browser specific features in the form of proprietary tags caused a fragmentation of the Web. Sites were tailored for a particular browser, and in the worst case ended up unreadable in another client. Bit 5.2
HTML is structural markup used mainly for presentation purposes
Common usage subverts even the structural intent of some tags, raising non-visual rendering issues. Consider the pervasive misuse of complex nested ‘table’ markup used simply as a layout expedient, or ‘blockquote’ just to achieve visual indentation. In HTML 4.0, and with the introduction of CSS, the purely visual elements were deprecated in the core markup. The intent was to achieve a new common standard for all browsers, with a clean separation of structural and visual markup (assuming consistent and proper application of the former). However, the basic premise and weakness remained of an externally defined and fixed set of tags, inflexible and ill-suited to serious metadata use.
Disentangling the Logic (XHTML and XML) The ad hoc entanglement of logical and visual markup in HTML documents was a major reason for the development of stricter, yet more flexible, markup languages for the Web. Currently in deployment, and supported by the newer Web clients, the best-known of these is
The Semantic Web
120
XML. It provides a much richer and more logical approach to incorporating metadata about the content. HTML, even considering just its structural-tag subset, can provide little meaningful metadata about the content apart from a few (and as noted, often inconsistently applied) logical markup elements. XML, on the other hand, allows for ‘meaningful’ and self-defining tags that can actually convey something specific about the nature of the tagged content – plus user-defined formats that are consistent. To define the tags, the document must contain a URI pointing to the resource where these definitions are found – an XML namespace, or possibly several. XML is actually a metalanguage; a language that can represent other languages in a standardized way. It is a popular framework for various other metadata applications, a situation that often (and to advantage) blurs the border between local application spaces and a wider Web context. Web Services can thus exchange information using a common extensible format. It is even commonplace to use XML for storing application registry values. Therefore, XML is an important component in the Semantic Web context. Bit 5.3
XML is a metalanguage markup framework, not just ‘another set of tags’
As seen in later discussions, XML can describe relationships and meaning, besides providing structural markup in a more consistent and flexible way than HTML. For the traditional Web at large, and importantly for the emerging Mobile Internet initiatives, the derived XHTML reformulation of HTML in the XML context is critical. Foremost, XHTML is the transitional vehicle for the predominantly HTML-tagged content on the Web today. It is the basis for the second incarnation of wireless application protocol (WAP2 or WML), which is essentially just XHTML Basic, a minimalist but strict version of XHTML (as noted later in the section on special-purpose markup). Figure 5.1 is a simple graphic representation of the relationship between the three Webmarkup languages mentioned here. As indicated, presentational markup (usually visual styling and layout) is separated and coded as CSS or XSL adjuncts. Stylesheet Extension While existing CSS can be used ‘as is’ with XML to specify presentational aspects of a document, the extensible albeit more complex XSL gives greater possibilities. Especially in
Figure 5.1 Relationship between HTML, XHTML and XML simplified and expressed as a graph depicting how XHTML is a subset of XML, recasting HTML in this more consistent framework
Languages and Protocols
121
the context of Semantic Web applications and the manipulation of content and content metadata, such capability offsets the overhead of XSL. In summary, using XSL invokes three parts: XSL Transformations (XSLT), which is a declarative language for transforming XML documents into other XML documents. Although XSLT can be used independently for arbitrary transformations, it is primarily intended for transformations relevant to XSL use. XML Path Language (XPath), which is an expression language used by XSLT to access or refer to parts of an XML document. This component, though not formally part of XSL, enables location-dependent functionality in applications. XPath is also used by the XML Linking specification, in XPointer, and in XML Query. XSL Formatting Objects (XFO), which is an XML vocabulary for specifying formatting semantics. This component expresses the main relevancy of XSL to the Semantic Web. A clarifying note on the terminology used here might be in order. When originally conceived, the whole was simply XSL, comprising both stylesheet and formatting. The effort was later split more formally into two halves: the transforming language (XSLT) and the formatting vocabulary (XFO). The focus on transformation in the first part led developers to use the shorter term XTL (XML Transformation Language) in this context. XTL is seen primarily as a conversion and transport protocol. Many developers tend to concentrate on XTL server-side transformations, from XML to display HTML or XHTML – in particular, to the more restrictive WML and WAP2 formats used in small mobile clients. The benefits of this approach are twofold: content can be published once and annotated in rich XML format, suitable for the more demanding applications, yet be customized on the fly to fit clients with limited capabilities. XLink is a separate standard (a W3C recommendation, see www.w3.org/TR/xlink/ ) for extensible crafting and embedding in XML documents of described links between resources. In their simplest forms, XLinks resemble HTML links. In more complex forms, they allow XML documents to assert linking relationships at both endpoints, assert relationships among more than two resources, associate metadata with the links, express links that reside in a location separate from the linked resources, and point to only specific internal portions of the destination resource. XML Pointer Language (XPointer) is an XLink implementation: the language to be used as a fragment identifier for any URI-reference that locates an XML-defined resource. One of its scheme parts is based on XPath. The overall design goal is to capture and preserve a richer semantic content in XML on the server, irrespective of the served output of the moment, usually metadata constrained. XML and the Semantic Web Revisiting the W3C-based diagram (Figure 2.5) introduced at the end of Chapter 2, we can at this stage view a different continuation of the relationship chain from ‘Agents’ to cover the relationship that XML has with the path to the goal, and thus how it fits into the overall scheme of the Semantic Web. A variation view is shown in Figure 5.2, extending the relationship chain into the lower levels of implementation can illustrate more detail of the markup space around the XML node and associated functionality.
The Semantic Web
122
Figure 5.2 A variation of the W3C diagram provides a different continuation of the relationship chain from Agents, now with a focus on XML’s markup space
We may note that most of the How? targets of ‘XML Enhancements’ (that is, components needed to implement it) are drawn as ‘planned’ because they have yet to be fully nailed down. Only XML has stabilized enough that it can be considered implemented. The arrow leading off to the left from XML implies a further continuation to be examined in a diagram in the next section.
Emerging Special-Purpose Markup Specialized markup is continually being developed, fortunately often based on XML or subsets of it. This development is especially true and important concerning the new categories of edge devices that will form a significant component in the new Web. Bit 5.4
Recent specialized markup is almost always based on XML
Communicating with the rest of the Web using a common protocol, or at least a subset or well-defined extension, ensures a solid basis for seamless interoperability. For this reason, Web markup that is not XML compliant is usually considered ‘obsolete’ today. Other reasons to develop new protocol variants include explicit support for new media types supported by the device – for example, mobile-phone/PDA multimedia. Compact Markup for Hand-held Devices Due to constraints om processor, memory, power, and display, full-sized markup support is neither practical nor even possible in small, hand-held devices. CHTML (Compact HTML, see www.w3.org/TR/1998/NOTE-compactHTML-19980209/) is a well-defined subset of HTML (2.0/3.2/4.0) recommendations, and is designed for small information appliances. Originally specified by the Japanese company Access Company Ltd., it was adapted for use with small computing devices such as the I-mode PDAs, cellular phones, and smart phones. CHTML excludes support for the more demanding features of
Languages and Protocols
123
HTML pages, such as tables, image maps, backgrounds, font varieties, frames, style sheets, and JPEG images. The intent is to enable small devices to connect to the open Web, rather than merely a customized section of it. Although CHTML is gracefully compatible with HTML documents on the Web, it must be considered obsolete for the same reasons as HTML. WML (Wireless Markup Language, www.wapforum.org/what/technical.htm) is then a better solution because it is based on XML, as a dialect specified by the WAP-forum to be used in poor network connections. It provides a complete set of network communication programs comparable to and supportive of Internet’s set of protocols. The main disadvantage of WML has been the lack of direct legacy-HTML support, thus denying the clients access to the larger portion of published Web content. This lack was addressed by reworking WML into v2 to support XHTML Basic, leveraging efforts to map HTML into XML. Multimedia Markup Legacy-HTML support may be one thing, but the current drive in both traditional Web publishing and hand-held devices (such as mobile phones) is to enable extensive multimedia support, including pervasive image capture (for example, phone-cams). This desire led to the development of special multimedia protocols based on XML. Bit 5.5
Extensible markup allows easy upgrade paths for new devices
Leveraging the transform capabilities of XTL, upgraded devices can use the same XMLbased markup to request higher-definition versions of the same content. MMS (Multimedia Messaging Service) is based on the XML vocabulary SMIL (described next) and standardized by the 3rd Generation Partnership Project (3GPP, www.3gpp.org) and the WAP Forum. Meant to be the successor to the popular Short Message Service (SMS) for mobile phones, for any device MMS can encapsulate and transmit multimedia presentations that contain images, sounds, text, and video clips. Multi-media in MMS documents are stored either as URI pointers to external resources, or as MIME-encoded blocks embedded in the document. SMIL (Synchronized Multimedia Integration Language, see www.w3.org/AudioVideo/ ) was developed by a group coordinated by the W3C, which included representatives from CD-ROM, interactive television, Web, and audio/video streaming industries. SMIL is designed to control complex multimedia presentation, and it enables Web site creators easily to define and synchronize multimedia elements (video, sound, still images) for Web presentation and interaction. Each media type is accessed with a unique URI/URL, which means that presentations can be composed using reusable media objects arriving from distributed stores. Presentations can also be defined as multiple versions, each with different bandwidth requirements. Numerous applications already implement extensive SMIL support, such as the popular media players and Web browsers, including the newest generations of mobile phones (which incorporate it in their MMS capability).
124
The Semantic Web
XUL/XML Straddling the fields of basic markup and functionality programming, the XML User Interface Language (XUL, main resource at www.xulplanet.com) is a recent language that figures prominently in the development of Web clients based on the Mozilla project’s Gecko rendering engine. Using XUL and other render components, developers create sophisticated applications and interfaces without special tools. Notable examples are the Firefox browser and Thunderbird e-mail client. The specific attraction of XUL is that it can be used in any context where you might use a Web application. It leverages portable XML-based specifications (and other XML-derived standards) and core components to enable richer user interfaces that can be customized to particular contexts, for example to retrieve and process resources from the network. In XUL you create an interface, use CSS style sheets to define appearance, and use JavaScript for client behavior. Programming interfaces support reading and writing to remote content over the network, calling Web services, and reading local files.
Higher-Level Protocols Markup provides the building blocks, but first when these blocks are used to define higherlevel protocols to accomplish some form of processing or message exchange, can we achieve a functionality greater than merely static representations of published content on the Web. Creating Web services without the proper messaging protocols and tools would mean much more hard work. Messaging (so-called higher transport) protocols were previously designed somewhat ad hoc, without much in common except their reliance on the same underlying transport protocols for the network. However, they are now typically based on or derivative of XML, and are essential to Web Services because they formalize a common framework for how services and applications are to interpret and process exchanged data. Figure 5.3, as a continuation of the previous diagram, illustrates how transport protocols fit into the overall scheme, and the way the relative dependencies and influences run. Markup and transport are the workhorses, and it is at these levels that we have hitherto seen most development – extending, refining, and harmonizing the capabilities of the
Figure 5.3 In this continuation of the previous diagram, we see how the focus shifts to transport protocols to convey information between endpoints. Here, we have largely backed into the existing preSW structures (the ultimate answers to How?), and away from the nebulous higher goal
Languages and Protocols
125
infrastructure of the Internet. In this context, we additionally need to consider systems for codifying a declaration of the content type, such as MIME. At minimum, we need to allow for mapping between locally defined namespaces that define how blocks of data and associated metadata are to be interpreted. The later sections on ontology and schema explain the approaches taken in this regard.
A Messaging Framework (SOAP versus REST) Brought to a greater public awareness because of its adoption in the. NET strategy, SOAP is an XML-based, ‘lightweight’ protocol for the exchange of information within decentralized, distributed environments. Its primary role is to expose existing application functionality as a WS, specifying API hooks for remote applications. The original name was Simple Object Access Protocol, but this full expansion is now officially deprecated due to the change in focus away from just ‘accessing objects’ towards a generalized XML messaging framework. In fact, the acronym itself might well change in future. The protocol was originally developed and implemented in 1999 by Dave Winer and was intended as an XML-RPC interface to the CMS package Manila (see www.userland.com). Manilla is one of the products that defined and popularized the now ubiquitous weblog (or ‘blog’) concept. However, that bit of protocol history now seems largely forgotten in the .NET context. SOAP is discussed most often in the HTTP and HTTP Extension Framework – that is, in the Web context. The practical implementation of SOAP-like protocols turns out to have great potential reach, as is often the case with simple designs. In effect, it puts XML back into the message-passing business exemplified by the original Internet, but this time at a new level. The intent is that applications and Web services employ the extension as the common framework for packaging and exchanging XML-tagged content and functionality. See it as a Web Services transport layer that can potentially be used in combination with a variety of other protocols. As it happens, SOAP is also an example of how XML can be used to define new protocols. We return somewhat later to a critique of how it has been implemented. We may note that e-business figures prominently in early example scenarios, mainly because SOAP provides an application-agnostic way to exchange typical transaction information. Characteristics and Specifications Since June 2003, SOAP v1.2 has had W3C recommendation status, compliant with the W3C XMLP requirements. It is specified at the W3C site (see version links from www.w3.org/ TR/ SOAP/ ), while general developers tend to visit SoapWare (www.soapware.org) for resources and the latest news. The three defining characteristics that satisfy the basic XMLP requirements are: Extensible. While simplicity remains a primary design goal, features expected in more demanding environment, such as security, routing, and reliability, can be added to SOAP as layered extensions.
126
The Semantic Web
Protocol independent. The SOAP framework is usable over a variety of underlying networking protocols, though it is commonly bound to HTTP. Model independent. SOAP is independent of programming models, as it just defines a way to process simple one-way messages. In the Microsoft .NET context, we additionally see reference to GXA (Global XML Web Services Architecture) and WSE (Web Services Extensions), which are initiatives to implement extensions to the base protocol. Microsoft has moved its main SOAP Web resource location occasionally (at time of last revision, links can be found at msdn.microsoft.com /webservices/ ). Variants of SOAP are implemented in several WS approaches, such as Axis (ws.apache.org/axis/ ) by the Apache Open Software Group. The aim here is to provide a reliable and stable base on which to implement Java Web Services (mentioned in Chapter 3). Sun’s J2EE platform, based on Java, implements a portfolio of protocols for WS. This work is portable to most platforms of sufficient capacity. As the API-bearing protocol names suggest, the developer focus is primarily on hacking the RPC-interfaces to services, not necessarily rethinking these services in URI contexts (see later discussion about REST). Current SUN strategy appears to involve interoperability with .NET. Whether the focus on API will ultimately limit applicability to SOAP-wrapping existing RPC-resources remains unclear. Functional Analysis Analyzing SOAP, we can say that it consists of three parts: An envelope that defines a framework for describing what is in a message and how to process it. A set of encoding rules for expressing instances of application-defined datatypes. A convention for representing remote procedure calls and responses. The envelope concept can be illustrated by depicting how a SOAP message is structured in its header and payload parts as part of an overall MIME-encoding. Figure 5.4 shows one such simple diagram of the encapsulation. The framework is the specification of how to parse the result – for example, in order to identify and locate payloads. The protocol specifications also define a complete processing model that outlines how messages are processed as they travel through a path between endpoints. In summary, there are two fundamental styles of SOAP messaging today: Document style indicates that the body contains an XML document. Sender and receiver must agree upon the format of this document. RPC style indicates that the body contains an XML representation of an encapsulated method call to the destination endpoint process.
Languages and Protocols
127
Figure 5.4 Schematic showing the internal structure of a MIME-encoded SOAP message, which in turn points to one or more payloads using the XML XLink pointer to provide necessary in-document locator information. The method is especially useful when sending non-XML payloads
Two techniques can be used to serialize the data into the body of the message: Schema definitions, which literally and unambiguously define the XML format for the body. This technique is most common for document style, and is called document/literal. SOAP encoding rules, which require the SOAP processor to walk through the various SOAP encoding rules at runtime to determine the proper serialization of the body. This technique, rpc/encoded, is more prone to errors and interoperability issues, but is commonly used in RPC messaging style. Document/literal is the preferred mode in the Web Services context. Rpc/encoded mode is commonly used when wrapping existing RPC-services instead of redesigning them to be schema oriented. Applications and Implementation Techniques The published implementations list (at www.soapware.org/directory/4/implementations) is long and slightly opaque. Closer inspection finds mainly projects to implement basic server support rather than any real WS applications. Revisiting the list in April 2005 finds no updates since January 2003, which makes it now seem rather dated. Other (now also dated) implementation examples may be found in the 2002 PerfectXML article ‘The XML Web’ (at www.perfectxml.com/artides/XML/TheXMLWeb.asp). Bit 5.6
The ‘great potential reach’ of SOAP remains somewhat elusive
Much of the reason for the seeming slow client-side adoption of SOAP must be ascribed to the general lack of deployed Web services on the public Web. SOAP is otherwise central to several architectural models of WS support, including .NET WS. RESTian Critique Developers have debated for years about whether SOAP, albeit simple in principle, is in fact an over-engineered, unnecessary addition to the WS infrastructure. The reasoning here is that the critics find very little ‘Web’ and much ‘API’ in the specifications.
128
The Semantic Web
At the center of this critique was Roy Fielding, chair of the Apache Software Foundation and co-writer of a number of the Web’s core specifications. In his 2000 doctoral thesis, he identified an architectural style called REST (Representational State Transfer) as the basis of the Web. It deals with resources by using URIs to exchange resource representations (mainly as documents), and a handful of core methods (HTTP GET and POST). Simplicity and generality are characteristic of this approach. Therefore, REST does not require you to install a separate tool kit to exchange WS data. The assumption is that everything you need to use WS is already available in the existing Web protocols if you know where to look. The strength and flexibility of the early Web (based on HTML, HTTP, and URI) rests on the single fact that the primitive HTTP natively supported URI notation. All methods being defined as operations on URI-addressed resources led to the viral-like success of the Web. Bit 5.7
REST is a demonstrated sound model for distributed computing
It is the model used by the world’s largest distributed computing application, the Web. The issue is largely about dereferencing the resources so that a user can concentrate on the ‘what’ rather than be distracted by ‘where’ and ‘how’. It also explains why search engines, despite their current flaws, became the dominant way to find things on the Web. When the user just has to submit a form query and click on a result link, and such forms can easily be incorporated on any Web page using the existing infrastructure, it is hard to beat – it embodies ‘the REST way’. In the words of Paul Prescod, in a 2002 article ‘REST and the Real World’ (webservices.xml.com/pub/a/ws/2002/02/20/rest.html), applying REST principles to WS-design is essential to the growth, innovation, and success of Web services: Unlike the current generation of Web services technologies, however, we make those three technologies central rather than peripheral, we rethink our Web service interface in terms of UPJs, HTTP, and XML. It is this rethinking that takes our Web services beyond the capabilities of the first generation technologies based on Remote Procedure Call APIs like SOAP-RPC.
Effectively adopting the opposing stance, one major WS stakeholder, Google, forged ahead with exposing SOAP APIs, at once promoting more complex and more constrained solutions. It also discontinued experiments with simple REST-style services for metadata (such as the ‘google.com/xml?q¼’ query). The RPC-oriented developer thus needs to ‘hack the interfaces’ to leverage the resources, hoping the site-specific APIs and URLs remain constant. This approach is not agent-friendly, and may require extensive documentation to figure out. It is also strongly technology dependent. Regardless of the justified critique, however, enterprise will adopt (and is adopting) SOAP-RPC mainly because it presents a low-cost, non-disruptive way to migrate the existing application infrastructure into the WS environment. Rethinking business process is not a priority.
Languages and Protocols
129
The approach of building complex API rules is sanctioned and promoted by the big players, so it will evidently always be with us to some extent. It is hoped it will not needlessly complicate, dominate, and restrict the Web resource technology. The technology you use to move bits from place to place is not important. The business-specific document and process modelling is . . . What we need are electronic business standards, not more RPC plumbing. Expect the relevant standards not to come out of mammoth software vendors, but out of industrial consortia staffed by people who understand your industry and your business problems. Paul Prescod
Bit 5.8 Migrating existing infrastructure into a WS framework has profound consequences ‘Virtually every software asset can now be offered, or soon will be able to be offered, as a Web service.’ Jon Udell, in 2002 (www.infoworld.com/artides/pl/xml/02/06/24/020624 plcomp.html). Slippery Security The decisive characteristic of Web services is that they must be designed to work across organizational boundaries, across arbitrary and poorly controlled networks. This constraint has serious implications with respect to security, auditing, and performance. Returning to REST, we might wonder what Web services look like when not constructed by exposing resource-server API structures and RPC-wrappers. Clearly, they too are based on a message protocol, and could well use SOAP, albeit in its more general document mode. However, the focus here is squarely on the network and identity, not the details of resource implementation. Where RPC protocols try hard to make the network look as if it is not there, such selective fudging has serious security consequences. You cannot deal with resource access across a network in the same way as when accessing from the desktop in the corporate LAN. Automatically passing incoming network messages as RPC method parameters directly to local resource software is risky, to say the least. Many RPC exploits are known, and they all tend to be serious. What the REST approach requires is to design a network interface in terms of URIs and XML resources. The usually object-oriented back-end implementation is therefore by necessity slightly distanced from the network side. Administrators can apply Access Control Lists (ACL) to secure services which use URIs as their organizational model, and to every URI-designated document that passes through the service. It is much harder to secure RPCbased systems where the addressing model is proprietary and expressed in arbitrary parameters. Each step in a business process is then represented by a new document with an attached ACL. All actors have a shared view of the URI-space representing the process. The URIs point to shared documents on the respective Web servers. Standard HTTPS/HTTP authentication and authorization are sufficient to keep intruders from also being able to look at the documents.
130
The Semantic Web
Note also that session-generated unique URI-designated documents provide an easy way to track, link, and preserve all transaction contexts. This mechanism both ensures that involved parties stay in sync with each other, and that security audit requirements are met. With regards to corporate firewalls, using plain HTTP is being very explicit about what is going on. By contrast, XML-RPC or SOAP on top of HTTP implements a tunnel, hiding the real actions in the XML body just to get through a firewall. Such obfuscation deliberately subverts security work and reduces the efficacy of audit and filter tools. Bit 5.9 Anything possible with SOAP or any RPC can be implemented with URIs and the four base methods in HTTP (GET, PUT, POST, and DELETE) As HTTP has only a few methods, developers can extend clients and servers independently without confusion and new security holes. Rather than invent new methods they find ways to represent the new concepts in XML data structures and headers. URI-centric Web services are inherently easy to integrate. The output from one service can form the direct input to another, merely by supplying its URL. That WS specifications such as SOAP and WSDL lack any real support for this existing, simple yet powerful, datasharing mechanism is a surprising discovery. Instead, they rely on complex translations to internal and implicit data models, ignoring the existing URI-resource of the Web and thus also many real-world requirements. Essential WS What HTTP needs to implement properly and fully secure WS, according to REST proponents, are concepts such as the following:
transaction rollback mechanisms for distributed services; explicit and implicit store and forward relays; transformational intermediaries; validated return paths (call-back, notification).
These features may be implemented as native sweb infrastructure regardless. The extensions may in fact be borrowed from SOAP headers and SOAP routing, representing perhaps the only part of SOAP that is strongly compatible with the Web and with HTTP. Ongoing discussions on REST can be found on RESTwiki (rest.blueoxen.net/cgi-bin/ wiki.pl), among other places. Community support is evidenced by signs that REST-style WS is what people actually use – for example, when Amazon ran both SOAP and REST interfaces, the latter handled in the order of 85% of all requests (REST requests were also six times faster to process). Implementing REST WS is also a less daunting task than to implement SOAP-RPC (the usual approach). Any site that presents a service process as a series of Web pages is easily modified to do the same thing with XML. Re-implementing the business logic is not required.
Languages and Protocols
131
However, REST is not a plug-in solution to everything. One common issue is that it requires you to rethink the application in terms of manipulations of addressable resources. The usual design approach today calls for method calls to a specified component. Server-side implementation is arbitrary and behind the scenes, to be sure, but the API you communicate to your clients should be in terms of HTTP manipulations on XML documents addressed by URIs, not in terms of method calls with parameters. Anyway, given any protocol for expressing messages, what shall the messages express?
Expressing Assertions (RDF) To codify the meaning of Web content – in other words, do more than simply mark up syntactical content elements – the Resource Description Framework was devised as a way to mark up metadata. More to the point, RDF provided a means to exchange metadata, and the concept is fundamental to later constructs. The defining elements or rules are: Resource, which is anything that can be assigned a URI (for example as a Web-location URL). Property, which is a named Resource that can be used as a property. Assertion, which is a statement about some relationship, as a combination of a Resource, a Property, and a value. Quotation, which is making an Assertion about other Assertions. Expressibility, in that there is a straightforward method for expressing these abstract Properties in XML. Assertions are like simple sentences in natural languages, the statement parts predictably denoted with the linguistic terms subject, predicate, and object – for example, ‘Resource x (URI) is/has Property y (URI) value z (optional URI)’. The common programming notion of key-value pairs (hash) can be recast as a triplet by understanding that the predicate is in this case implicit in the context. RDF triplets can occur in an arbitrary order – something that seems natural in the context of metadata. By contrast, element order can be very significant in XML. Figure 5.5 provides a visual illustration of such three-part assertions.
Figure 5.5 Example of simple RDF triplets applied to a typical case, asserting two properties for a book-type resource. A real-world example would have many properties, not just two. Libraries are big users of RDF
132
The Semantic Web
Figure 5.6 An example of a simple RDF triplet when a value is itself a resource about which an assertion can be made. Further indirection is also possible in that any value can be a resource
A concrete simple assertion using a resource variable might be ‘MyLanIP is 192.168.1.3’ – or as URIs for each element of the triplet:
Figure 5.6 builds on the previous example to show how indirection results when a value is itself defined as a resource about which assertions can be made. Implied in the diagram is that the properties are defined in a common storage location given for the particular metadata application, but while common in practice, this assumption is not necessarily true. Each property (as resource) is associated with its own URI. RDF is carefully designed to have the following characteristics: Independence, where as a Resource, any independent organization (or person) can invent a Property which can be properly referenced and interpreted by others. Interchange, which is expedited by conversion into XML, the new lingua franca of Web exchange. Scalability, where simple, three-part RDF records (of Resource, Property, and value) are easy to handle and use as references to other objects. Properties are Resources, which means that they can have their own properties and can be found and manipulated like any other Resource. Values can be Resources, which means they can be referenced in the same way as Properties, have their own Properties, and so on. Statements can be Resources, which allows them to have Properties as well – essential if we are to be able to do lookups based on other people’s metadata and properly evaluate the Assertions made. In passing, we might note that since properties and values can also be resources, they can be anything represented in a resource, not just words (although the latter is an understandable assumption made by most people).
Languages and Protocols
Bit 5.10
133
RDF is a way of exchanging ‘objects’ rather than explicit elements
Business applications in particular find advantage in RDF because of the way business data often have complex structure best described as objects and relations.
The Containment Issue It has been argued that RDF-element order is not totally arbitrary, but rather that there is some significance in an order-related ‘containment’ concept. Perhaps so, but it turns out to be elusive to capture, despite the fact that containment seems significant to human readers. Why should ‘RDF containment’ be elusive? The answer is perhaps surprising, but RDF structures do not formally need the concept at all. A deeper analysis suggests that containment is significant only for the ‘delete data’ operation, which in any case many purists eschew on principle. The ‘correct’ way to amend data is to annotate, introduce new pointers, but never remove. The containment concept turns out to seem relevant only because of the intuitive human understanding that deleting some object also deletes everything ‘contained in that object’. Aha! We find that it is ultimately an implied, human-semantic interpretation.
RDF Application By itself, RDF is simply an outline model for metadata interchange, defining a basic syntax in which to express the assertion relationships that make up the metadata. In fact, it is such a simple model that initially it takes a considerable amount of work to implement anything usable. But once defined, the broad capability for reuse makes the effort worthwhile. Simplicity aside, an automated RDF Validation Service is provided by the W3C (found at www.w3.org/RDF/Validator/). Formal validation is always recommended whenever markup is involved – whether (X)HTML, XML, or RDF. Raw tagged text tends to be opaque even to experienced readers, and subsequent machine parsing is always highly intolerant of syntactical errors. Application of RDF generally takes the form of creating XML templates for particular cases, where the resources and properties are structured in the assertions that will define the relationships to be mapped from the available data in some database. The XML template is constructed as an abstract model of metadata, and it may also contain information about how to view and edit data in the model. Populating the template with actual values from a database record generates a viewable document to display as a response to an end-user request. Figure 5.7 illustrates a typical flow around a metadata database using populated templates.
Using RDF Vocabularies Properties would normally not be defined in isolation, but would instead come in vocabularies – that is, ready-to-use ‘packages’ of several context-associated descriptors.
134
The Semantic Web
Figure 5.7 A possible application loop using RDF XML templates to access and manage metadata from a database. This model is fairly representative of many metadata management implementations
As an example, a set of basic bibliographic Properties for a simple book database might be Author, Title, and Publication Date, which in a more formal context can be extended with ISBN, Publisher, Format, and so on. Different vocabularies would be expected to arise for any number of Web contexts. The nice thing about RDF vocabularies is that correspondences between similar vocabularies can easily (and globally) be established through new Assertions, so that one set’s Author can be related to another’s Creator. As anyone can create vocabularies, so anyone can create and publish correspondence definitions. It is unrealistic to think that everyone will (or can) use the same vocabulary; that is neither desirable nor necessary. Bit 5.11
Custom RDF vocabularies can become valuable commodities
We can envision that opinions, pointers, indexes, and anything that helps people discover things on the ever-growing Web can become commodities of very high value. With RDF (and appropriate tools) anyone can invent vocabularies, advertise them, and sell them – creating a free marketplace in vocabularies. Most niches of information would probably come to be dominated by a small number of specialized vocabularies, determined mainly by consensus and usage – or, if you will, by marketing and market-share. The really useful vocabularies would represent investments of expertise comparable to typeface designs, for example, or good technical dictionaries. A ballpark estimate on the number of uniquely referenced objects on the Web today is in the order of half a million. It would clearly be a wasteful effort for everyone to reinvent vocabularies representing relationships between them all. Yet such wastefulness is precisely what happens when proprietary KB systems are implemented, and it risks happening if future RDF vocabularies are not made open, available for free or for modest licensing. RDF is agnostic in terms of processing software. Different software implementations can process the same metadata with consistent results, and the same software could process (at least in part) many different metadata vocabularies.
Languages and Protocols
135
Extended RDF Layers As defined, RDF is both general and simple. The language at this point has in fact neither negation nor implication, and is therefore very limited. Consequently, further layers are required to implement a broader and more complex functionality than simply making statements about properties and statements about other statements. Fortunately for early adoption, applications at this basic level of RDF are still very numerous. Such applications focus on the representation of data, which typically involves simple operations such as indexing, information storage, labeling, and associating style sheets with documents. This kind of processing does not require the expression of queries or inference rules. While RDF documents at this level do not have great power, it is expected that these simple data will eventually be combined with data from other Web applications. Such extensions would require a common framework for combining information from all these applications – reason enough to go to the trouble of already now RDF-mapping the data in the simple applications. Bit 5.12
Deploying RDF structure early is a way of evolving towards future reuse
Whether the local applications at present require RDF is immaterial, because once the data are mapped, new application areas for existing data become possible (and profitable) in ways that are hard to foresee. The power of the greater whole will then greatly surpass what is possible in the local data representation language. Powerful query or access control logic implemented elsewhere can reference the local data through RDF exchanges. Even though it’s simple to define, RDF . . . will be a complete language, capable of expressing paradox and tautology, and in which it will be possible to phrase questions whose answers would to a machine require a search of the entire Web and an unimaginable amount of time to resolve. Tim Berners-Lee
Referencing Namespaces XML makes the (re)use of predefined vocabularies, such as within custom RDF schemas (described later), fairly straightforward with its namespace concept. This capability has particular application when converting from one namespace to another. Bit 5.13
RDF can correlate different properties that have similar meanings
The choice of a tag name depends on local usage and intended purpose, but the values assigned to one property can often fit for another property in another context. In this way, an application can import appropriate externally-maintained values. For example, suppose that a particular e-commerce implementation uses the tag shipTo, but could benefit from directly referencing another resource’s definition of Address. The
The Semantic Web
136
advantage would be that the former is then directly tied to any changes in the latter, which might reside in a central database for customer records. Such referencing utilizes two steps: A namespace declaration, which introduces (‘imports’) the other schema to which reference is being made. A type reference, which embodies the actual reference to the other resource’s stored value. The namespace declaration in XML consists of an attribute ‘xmlns:’ that defines a relation between a local prefix used in naming tags and an external schema referenced by a URI (often as a Web URL, and then often using a persistent PURL pointer): xmlns: wherestuff ¼"http:// where.com / stuff .xml"
The URI is the exact reference, which is tied to the arbitrary prefix that can be used locally to consistently reference the remote schema in the rest of the XML document. The external schema types can thereafter be used almost as if they were defined in the local schema:
While ‘Address’ alone would uniquely reference a name in the local schema, each prefixed version is a different unique reference to an item in a another URI-specified schema. A practical schema typically leverages at least several external references, starting with the basic W3C schema (or equivalent) for defining the terms used to define the tags of interest. Element referencing is simple (and human-readable despite the many tags), as shown by a really simple example RDF file: Bo Leuf Information page for book The Semantic Web with support page links.
This snippet of metadata imbues a resource (the specified Web page) with two ‘about’ properties: creator and subject. Published at some public location on the Web, it thus makes assertions about these properties for the resource.
Languages and Protocols
137
The meaning of the ‘about’ term is defined in an external resource maintained by W3C, and referenced after the xmnls declaration by the prefix in the ‘rdf:about’ container. The creator and subject properties are standard elements defined by a Dublin Core v1.1 (‘dc:’) reference that was created for publication metadata. This DC base list contains the 15 most relevant properties for this purpose: Title, Creator, Subject, Description, Publisher, Contributor, Date, Type, Format, Identifier, Source, Language, Relation, Coverage, and Rights. Of course, other application areas can leverage these properties for other purposes, simply by referencing the DC definitions. RDF Schema The next layer up from just mapping the data into new formats is a schema layer in which to make meta-Assertions – statements about the language itself. For example, at this next level we can do a number of useful things, such as: declare the existence of a new Property; constrain the way a Property is used, or the types of object to which it can apply; perform rudimentary checks on Resources and associated Property values. This layer is formally adequate for dealing with conversions between different schemas, even though it too lacks logic tools, because most conversion is merely a matter of identifying and mapping similar Properties. Bit 5.14
A schema is both a defining and a mapping description
A schema describes types of objects and types of properties (attributes or relations), both of them being organized in two hierarchies of types. The full power of a schema layer, however, will ultimately depend on yet another layer: the logical layer, which can fully express complex logical relations between objects. Because of the self-defining nature of the XML/RDF edifice, the logic needs to be written into the very documents to define rules of deduction and inference between different types of documents, rules for query resolution, rules for conversion from unknown to known properties, rules for self-consistency, and so on. The logical layer thus requires the addition of predicate logic and quantification, in that order, to expand the simple statements possible in the previous layer. RDF Schema Sample Despite the wordy length, a selection of simple RDF schema samples might illustrate these conventions in their proper context. At the core is the W3C syntax specification, often referenced using the suggestive local prefix ‘rdf:’ when applying its terms:
The Semantic Web
138
This basic set of defined relational properties comprises just ten: Statement, Property, Bag, Seq, Alt, predicate, subject, object, type, and value. The container comment provides an
Languages and Protocols
139
informal definition for human readers (that is, programmers) who are setting up correspondences in other schemas using these standard items. A schema for defining basic book-related properties is found in a collection of Dublin Core draft base schemas (it is named dc.xsd at www.ukoln.ac.uk/metadata/dcmi/dcxml/ examples.html). It renders several pages in this book’s format, so it is listed for reference in Appendix A. A so-called ‘Simple DC’ application (using only the 15 elements in the ‘purl.org/dc/elements/1.1/namespace’) might import only this basic schema, yet be adequate for many metadata purposes. The namespace definitions provide the information required to interpret the elements. An excerpt from the ‘purl.org/dc/elements/1.1/namespace’, including the first property-container (for ‘title’, which follows the initial namespace description container) should be sufficient to show the principle: ]> The Dublin Core Element Set v1.1 namespace providing access to its content by means of an RDF Schema The Dublin Core Metadata Initiative The Dublin Core Element Set v1.1 namespace provides URIs for the Dublin Core Elements v1.1. Entries are declared using RDF Schema language to support RDF applications. English l999-07-02 2003-03-2 4
The Semantic Web
140
Title A name given to the resource. Typically, a Title will be a name by which the resource is formally known. 1999-07-02 2002-10-0 4 ...
This entry formally defines the DC Property ‘title’, with description and pointers to further resources that provide usage guidelines. Each of the other properties has a corresponding entry in the complete definition file. The previous examples show some variation in style and might seem off-putting in their length in relation to what they provide. The important point, however, is that they already exist as a Web resource. You do not have to write them all over again – just reference the resources and properties as needed in application RDF constructions. In any case, most of the tag verbosity in RDF definitions is generated automatically by the tools used to construct the RDF. How-to Adding RDF metadata to existing Web pages is actually rather simple. An example is given in Appendix A, right after the dc.xsd book schema mentioned earlier. The points to note are the addition of the about.xrdf file to establish provenance, and metadata blocks to affected existing pages. Optional steps leverage public-key technology to
Languages and Protocols
141
Figure 5.8 Conceptual overview of how client software might parse a served document into XML and then extract RDF information. The logic part parses and processes the RDF assertions perhaps to arrive at a new URI, which dereferenced becomes another Web request to a server
secure and validate content: a reference to a public key stored on the site, and digital signing of each page using the corresponding private key, storing the result as referenced signature files. Web Loops Putting XML and RDF into the loop, as it were, we can provide a sketch of how the Web and Semantic parts can intermesh and work together, shown in Figure 5.8. This sketch is illustrative only at the conceptual level. The Web part proceeds from an initial URI and sends a request (using the HTTP GET method), which is turned into a transfer representation of a document by the responding server – a returned stream of bits with some MIME type. This representation is parsed into XML and then into RDF. The Semantic part can parse the RDF result (graph or logical formula) and evaluate the meaning of statements contained in the document. It might then apply weighting based on signed keys, trust metrics, and various other rules, before selecting a course of action. The result might lead to the dereferencing of one or more other URIs, returning to the Web part for new requests. Application can be varied. For example, consider how someone might be granted access to a Web resource in the Semantic Web. A document can be issued which explains to the Web server why that particular person should have access. The underlying authority and authentication would then rest on a chain of RDF assertions and reasoning rules, involving numerous different resources and supported by pointers to all the required information. Distributed logic and agency at work! The availability of strong public key encryption and digital signatures is one of the factors that can make this kind of decentralized access control viable. Trust processing and signed digital certificates implement the Web versions of access keys and letters of authority.
The Graphical Approach As XML evolved to become the standard format of exchanging data, some developers found the RDF approach to metadata somewhat cumbersome to implement.
The Semantic Web
142
Bit 5.15
RDF has a useful data model but a problematic XML syntax
This view is said to be a widely held one within the XML developer community, one claimed to raise the threshold to achieving practical implementations. Their main objection was that heterogene