OPTICAL FIBER TE LEC 0 M MU N ICAT I SYSTEMS AND IMPAIRMENTS
OPTICAL FIBER TELECOMMUNICATIONS IV B SYSTEMS AND IMPAIRM...
141 downloads
5425 Views
28MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
OPTICAL FIBER TE LEC 0 M MU N ICAT I SYSTEMS AND IMPAIRMENTS
OPTICAL FIBER TELECOMMUNICATIONS IV B SYSTEMS AND IMPAIRMENTS
OPTICAL FIBER TELECOMMUNICATIONS IV B SYSTEMS AND IMPAIRMENTS
Edited by
IVAN P.KAMINOW Bell Laboratories (retired) Kaminow Lightwave Technology Holmdel, New Jersey
TINGYE LI AT&T Labs (retired) Boulder, Colorado
ACADEMIC PRESS An Elsevier Science Imprint San Diego San Francisco New York Boston London Sydney Tokyo
This book is printed on acid-free paper. @ Copyright @ 2002, Elsevier Science (USA). All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system,without permission in writing from the publisher. The appearance of the code at the bottom of the h s t page of a chapter in this book indicates the Publisher’s consent that copies of the chapter may be made for personal or internal use of speci6c clients. This consent is given on the condition, however, that the copier pay the stated per copy fee through the Copyright Clearance Center, Inc. (222 Rosewood Drive, Danvers, Massachusetts 01923), for copying beyond that permitted by Sections 107 or 108 of the U.S. Copyright Law. This consent does not extend to other kinds of copying, such as copying for general distribution, for advertising or promotional purposes, for creating new collective works, or for resale. Copy fees for pre-2001 chapters are as shown on the title pages. If no fee code appears on the title page, the copy fee is the same as for current chapters. $35.00.
Academic Press An Elsevier Science Imprint 525 B Street, Suite 1900, San Diego, California 92101-4495, USA http://www.academicpress.com
Academic Press Harcourt Place, 32 Jamestown Road,London NW17BY, UK http://www.academicpress.com Library of Congress Control Number: 2001098830 International Standard Book Number: 0-12-395173-9 PRINTED IN CHINA 02 03 04 05 06
07
RDC
9
8
7
6
5
4
3
2
1
For Florence and Edith, with love
Contents
Contributors
xi
Chapter 1 Overview
1
Ivan 19 Kaminow
Chapter 2 Growth of the Internet
17
Kerry G. Coflman and Andrew M. Odbzko
Chapter 3 Optical Network Architecture Evolution
57
John Strand
Chapter 4 Undersea Communication Systems
154
Neal S. Bergano
Chapter 5 High-Capacity, Ultra-Long-Haul Networks
198
John Zyskind, Rick Bany, Graeme Pendock, Michael Cahill, and Jinendra Ranka
Chapter 6 Pseudo-Linear Transmission of High-speed TDM Signals: 40 and 160 Gb/s Red-Jean Essiambre, Gregory Raybon, and Benny Mikkelsen
vii
232
viii
Contents
Chapter 7 Dispersion-Managed Solitons and Chirped Return to Zero: What Is the Difference?
305
Curtis R. Menyuk, Gary M. Carter; WilliamL. Kath, and Ruo-Mei Mu
Chapter 8 Metropolitan Optical Networks
329
Nasir Ghani, Jin-B Pan, and Xin Cheng
Chapter 9 The Evolution of Cable TV Networks
404
Xiaolin Lu and OIeh Sneizka
Chapter 10 Optical Access Networks
438
Edward Harstead and Pieter H. van Heyningen
Chapter 11 Beyond Gigabit: Application and Development of High-speed Ethernet Technology
514
Cedric E Lam
Chapter 12 Photonic Simulation Tools
564
Arthur J. Lowery
Chapter 13 Nonlinear Optical Effects in WDM Transmission
61 1
Polina Bayvel and Robert Killey
Chapter 14 Fixed and Tunable Management of Fiber Chromatic Dispersion
642
Alan E. Willner and Bogdan Hoanca
Chapter 15
Polarization-ModeDispersion
Herwig Kogelnik, Robert M. Jopson, and Lynn E. Nelson
725
Contents
Chapter 16
Bandwidth-Efficient Modulation Formats for Digital Fiber Transmission Systems
ix
862
Jan Conradi
Chapter 17
Error-Control Coding Techniques and Applications
902
E! Vjay Kumar, Moe Z. Win, Hsiao-Feng Lu, and Costas N. Georghiades
Chapter 18
Equalization Techniques for Mitigating Transmission Impairments
965
Moe Z. Win, Jack H. Winters, and Giorgio M. Etetta
Index to Volumes IVA and IVB
999
Contributors D. A. Ackerman (A:587), Agere Systems, 600 Mountain Avenue, Murray Hill, New Jersey 07974 Daniel Y. Al-Salameh (A295), JDS Uniphase Corporation, 100 Willowbrook Road, Bldg. 1,Freehold, New Jersey 07728-2879 Rick Barry (B: 198), Sycamore Networks, 10 Elizabeth Drive, Chelmsford, Massachusetts 01824-4111 Polina Bayvel (B:61 l), Optical Networks Group, Department of Electronic and Electrical Engineering, University College London (UCL), Torrington Place, London WCl E 7JE, United Kingdom Neal S. Bergano (B: 154), Tyco Telecommunications,250 Industrial Way West, Eatontown, New Jersey 07724-2206 Lee L. Blyler (A:17), OFS Fitel, LLC, 600 Mountain Avenue, Murray Hill, New Jersey 07974 Raymond K. Boncek (A: 17), OFS Fitel, LLC, 600 Mountain Avenue, Murray Hill, New Jersey 07974 Michael Cahil (B: 198), SycamoreNetworks, 10 Elizabeth Drive, Chelmsford, Massachusetts 01824-4111 Gary M. Carter (B:305), Computer Science and Electrical Engineering Department, TRC-201A, University of Maryland Baltimore County, 1000 Hilltop Circle, Baltimore, Maryland 21250 and Laboratory for Physical Sciences, College Park, Maryland Connie J. Chang-Hasnain (A:666), Department of Electrical Engineering and Computer Science, University of California, Berkeley, California 94720 and Bandwidth 9 Inc., 46410 Fremont Boulevard, Fremont, California 94538 Young-Kai Chen (A:784), Lucent Technologies, High Speed Electronics Research, 600 Mountain Avenue, Murray Hill, New Jersey 07974 Xin Cheng (B:329), Sorrento Networks Inc., 9990 Mesa Rim Drive, San Diego, California 9212 1-2930
xi
xu
Contributors
Dominique Chiaroni(A:732), Alcatel Research & Innovation, Route de Nozay, F-9 1461 Marcoussis cedex, France Kerry G. Coffman (B:17), AT&T Labs-Research, A5-1D03, 200 Laurel Avenue South, Middletown, New Jersey 07748 Jan Conradi (B:862), Director of Strategy,Corning Optical Communications, Corning Incorporated,MP-HQ-Wl-43, One River Front Plaza, Corning, New York 14831 Santanu J L Das (A:17), OFS Fitel, LLC, 600 Mountain Avenue, Murray Hill, New Jersey 07974 Emmanuel Desurvire (A:732), Alcatel Technical Academy, Villarceaux, F-9 1625 Nozay cedex, France David J. DiGiovanni (A:17), OFS Fitel, LLC, 600 Mountain Avenue, Murray Hill, New Jersey 07974 Christopher R Doerr (A:405), Bell Laboratories, Lucent Technologies, 791 Holmdel-Keyport Road, Holmdel, New Jersey 07733 Adam Ellison (A:80), Corning, Inc., SP-FR-05, Corning, New York 14831
L. E. Eng (A:587), Agere Systems, Room 2F-204, 9999 Hamilton Blvd., Breinigsville, Pennsylvania 18031-9304 Turan Erdogan (A:477), Semrock, Inc., 3625 Buffalo Road, Rochester, New York 14624 RenkJean Essiambre (B:232), Bell Laboratories, Lucent Technologies, 79 1 Holmdel-KeyportRoad, Holmdel, New Jersey 07733 Costas N. Georghiades (B:902), Texas A&M University, Electrical Engineering Department, 237 Wisenbaker, College Station, Texas 77843-3128 Nasir Ghani (B:329), Sorrento Networks Inc., 9990 Mesa Rim Drive, San Diego, California 92121-2930 Steven E. Golowich (A:17), Bell Laboratories, Lucent Technologies, Room 2C-357,600 Mountain Avenue, Murray Hill, New Jersey 07974 Christoph S. Harder (A:563), Nortel Networks Optical Components, Binzstrasse 17, CH-8045 Zurich, Switzerland Edward Harstead (B:438), Bell Laboratories, Lucent Technologies, 101 Crawford Corners Road, Holmdel, New Jersey 07733 Bogdan Hoanca (B:642), Phaethon Communications, Inc., Fremont, California 96538
Contributors
xiii
J. E. Johnson (A587), Agere Systems, 600 Mountain Avenue, Murray Hill, New Jersey 07974 Robert M. Jopson (B:725), Crawford Hill Laboratory, Bell Laboratories, Lucent Technologies, 79 1 Holmdel-Keyport Road, Holmdel, New Jersey 07733 Ivan P. Kaminow (A: 1,B: l), Bell Laboratories (retired), Kaminow Lightwave Technology, 12 Stonehenge Drive, Holmdel, New Jersey 07733 Bryon L. Kasper (A:784), Agere Systems, Advanced Development Group, 4920 Rivergrade Road, Irwindale, California 91706-1404 William L. Kath (B:305), Computer Science and Electrical Engineering Department, University of Maryland Baltimore County, 1000 Hilltop Circle, Baltimore, Maryland 21250 and Applied Mathematics Department, Northwestern University, 2145 Sheridan Road, Evanston, Illinois 60208-3125 L. J. P. Ketelsen (A:587), Agere Systems, 600 Mountain Avenue, Murray Hill, New Jersey 07974 P. A. Kiely (A:587), Agere Systems, 9999 Hamilton Blvd., Breinigsville, Pennsylvania 18031-9304 Robert Killey (B:61 l), Optical Networks Group, Department of Electronic and Electrical Engineering, University College London (UCL), Torrington Place, London WClE 7JE, United Kingdom Herwig Kogelnik (B:725), Crawford Hill Laboratory, Bell Laboratories, Lucent Technologies, 79 1 Holmdel-Keyport Road, Holmdel, New Jersey 07733 StevenK Korotky (A295), Bell Laboratories, Lucent Technologies,Room HO 3C-351,101 Crawfords Corner Road, Holmdel, New Jersey 07733-1900 P. Mjay Kumar (B:902), Communication Science Institute, Department of Electrical Engineering- Systems,University of Southern California, 3740 McClintock Avenue, EEBSOO, Los Angeles, California 90089-2565 and Scintera Networks, Inc., San Diego, California Cedric E Lam (B:514), AT&T Labs-Research, 200 Laurel Avenue South, Middletown, New Jersey 07748 Bruno Lavigne (A:732), Alcatel CIT/ Research & Innovation, Route de Nozay, F-91461 Marcoussis cedex, France Olivier Leclerc (A732), Alcatel Research & Innovation, Route de Nozay, F-91460 Marcoussis cedex, France
xiv
Contributors
David S. Levy (A295), Bell Laboratories, Lucent Technologies, Room HO 3B-506, 101 Crawfords Corner Road, Holmdel, New Jersey 07733-3030 Arthur J. Lowery (B:564), VPIsystems Inc., Design Center Group, 17-27 Cotham Road, Kew, Melbourne 3101, Australia Xiaolin Lu (B:404), Morning Forest, LLC, 8804 S. Blue Mountain Place, Highlands Ranch, Colorado 80126 Hsiao-Feng Lu (B:902), Communication Science Institute, Department of ElectricalEngineering- Systems, University of Southern California, 3740 McClintock Avenue, EEBSOO, Los Angeles, California 90089-2565 Amaresh Mahapatra (A:258), Linden Corp., 10 Northbriar Road, Acton, Massachusetts 01720 T. G. B. Mason (A:587), Agere Systems, 9999 Hamilton Blvd., Breinigsville, Pennsylvania 18031-9304
Curtis R. Menyuk (B:305), Computer Science and Electrical Engineering Department, TRC-201A, University of Maryland Baltimore County 1000 Hilltop Circle, Baltimore, Maryland 21250 and PhotonEx Corporation, 200 MetroWest Technology Park, Maynard, Massachusetts 0 1754 Benny Mikkelsen (B:232), Mintera Corporation, 847 Rogers Street, One Lowell Research Center, Lowell, Massachusetts 01852 John Minelly (A:80), Corning, Inc., SP-AR-02-01, Corning, New York 14831 Osamu Muuhara (A:784), Agere Systems, Optical Systems Research, 9999 Hamilton Blvd., Breinigsville, Pennsylvania 18031 Stefan Mohrdiek (A:563), Nortel Networks Optical Components, Binzstrasse 17, CH-8045 Ziirich, Switzerland Ruo-Mei Mu (B:305), Tyco Telecommunications, 250 Industrial Way West, Eatontown, New Jersey 07724-2206 Edmond J. Murphy (A:258), JDS Uniphase, 1985 Blue Hills Avenue Ext., Windsor, Connecticut 06095 Timothy 0. Murphy (A:295), Bell Laboratories, Lucent Technologies, Room HO 3D-516, 101 Crawfords Corner Road, Holmdel, New Jersey 077333030 Lynn E. Nelson (B:725), OFS Fitel, Holmdel, New Jersey 07733 Andrew M. Odlyzko (B:17), University of Minnesota Digital Technology Center, 1200 Washington Avenue S., Minneapolis, Minnesota 55415
Contributors
xv
Jin-Yi Pan (B:329), SorrentoNetworks Inc., 9990 Mesa Rim Drive, San Diego, California 92121-2930 Sunita 18. Pate1 (A:295), Bell Laboratories, Lucent Technologies, Room HO 3D-502,101 Crawfords Comer Road, Holmdel, New Jersey 07733-3030 Graeme Pendock (B: 198), Sycamore Networks, 10 Elizabeth Drive, Chelmsford, Massachusetts 01824-4111 Jinendra Ranka (B: 198), Sycamore Networks, 10 Elizabeth Drive, Chelmsford, Massachusetts 01824-4111 Gregory Raybon (B:232), Bell Laboratories, Lucent Technologies, 79 1 Holmdel-Keyport Road, Holmdel, New Jersey 07733 Gaylord W. Richards (A:295), Bell Laboratories, Lucent Technologies,Room 6L-219,2000 Naperville Road, Naperville, Illinois 60566-7033 Karsten Rottwitt (A:213), Orsted Laboratory, Niels Bohr Institute, University of Copenhagen, Universitetsparken 5, Copenhagen dk 2 100, Denmark Bertold E. Schmidt (A:563), Nortel Networks Optical Components, Binzstrasse 17, Ch-8045 Zurich, Switzerland Oleh Sniezko (B:404), Oleh-Lightcom, Highlands Ranch, Colorado 80126 Leo H. Spiekman (A:699), Genoa Corporation, Lodewijkstraat 1A, 5652 AC Eindhoven, The Netherlands Atul K. Srivastava (A:174), Onetta Inc., 1195 Borregas Avenue, Sunnyvale, California 94089 Andrew J. Stentz (A:213), Photuris, Inc., 20 Corporate Place South, Piscataway, New Jersey 08809 John Strand (B:57), AT&T Laboratories, Lightwave Networks Research Department,RoomA5-106,200LaurelAvenue, Middletown, New Jersey 07748 Thomas A. Strassser (A:477), Photuris Inc., 20 Corporate Place South, Piscataway, New Jersey 08854 Yan Sun (A:174), Onetta Inc., 1195 Borregas Avenue, Sunnyvale, California 94089 Eric S. Tentarelli (A:295), Bell Laboratories, Lucent Technologies, Room HO 3B-530,101 Crawfords Corner Road, Holmdel, New Jersey 07733-3030 Pieter H. van Heyningen (B:438), Lucent Technologies NL, PO. Box 18, Huizen 1270AA,The Netherlands
xvi
Contributors
Giorgio M. Vitetta (B:965), University of Modena and Reggio Emilia, Department of Information Engineering, Via Vignolese 905, Modena 41100, Italy W. White (A:17), OFS Fitel, LLC, 600 Mountain Avenue, Murray Hill, New Jersey 07974 Alan E. Willner (B:642), University of Southern California, Los Angeles, California 90089-2565 Moe Z. Win (B:902, B:965), AT&T Labs-Research, Room A5-1D01, 200 Laurel Avenue South, Middletown, New Jersey 07748-1914 Jack H. Winters (B:965), AT&T Labs-Research, Room 4-147, 100 Schulz Drive, Middletown, New Jersey 07748-1914 Martin Zirngibl (A:374), Bell Laboratories, Lucent Technologies, 79 1 Holmdel-KeyportRoad, Holmdel, New Jersey 07733-0400 John Zyskind (B: 198), Sycamore Networks, 10 Elizabeth Drive, Chelmsford, Massachusetts 0 1824-4111
Ivan P.Kaminow Bell Laboratories (retired),Kaminow Lightwave Technology, Holmdel, New Jerscy
Introduction Modern lightwave communications had its origin in the first demonstrations of the laser in 1960. Most of the early lightwave R&D was pursued by established telecommunications company labs (AT&T, NTT, and the British Post Office among them). By 1979, enough progress had been made in lightwave technology to warrant a book, Optical Fiber Telecommunications(OFlJ, edited by S. E. Miller and A. G. Chynoweth, summarizing the state of the art. Two sequels have appeared: in 1988, OFT 11, edited by S. E. Miller and I. P. Kaminow, and in 1997, OFT 111 (A & B), edited by I. P. Kaminow and T. L. Koch. The rapid changes in the field now call for a fourth set of books, OFTW (A & B). This chapter briefly summarizes the previous books and chronicles the remarkably changing climates associated with each period of their publication. The main purpose, however, is to summarize the chapters in OFT IV in order to give the reader an overview.
History While many excellent books on lightwave communications have been published, this series has developed a special character, with a reputation for comprehensiveness and authority, because of its unique history. Optical Fiber Telecommunications was published in 1979, at the dawn of the revolution in lightwave telecommunications. It was a stand-alone work that aimed to collect all available information on lightwave research. Miller was Director of the Lightwave Systems Research Laboratory and, together with Rudi Kompfner, the Associate Executive Director, guided the system research at the Crawford Hill Laboratory of AT&T Bell Laboratories; Chynoweth was an Executive Director in the Murray Hill Laboratory, leading the optical fiber research. Many groups were active at other laboratories in the United States, Europe, and Japan. OFT, however, was written exclusively by Bell Laboratories authors, who nevertheless aimed to incorporate global results. 1 OPTICAL FIBER TELECOMMUNICATIONS, VOLUME IVB
Copyright 0 2002, Elsevier Science (USA). All rights of reproduction in any form reserved. ISBN 0-12-395173-9
2
Ivan P. Kaminow
Miller and Chynoweth had little trouble finding suitable chapter authors at Bell Labs to cover practically all the relevant aspects of the field at that time. Looking back at that volume, it is interesting that the topics selected are still quite basic. Most of the chapters cover the theory, materials, measurement techniques, and properties of fibers and cables (for the most part, multimode fibers). Only one chapter covers optical sources, mainly multimode AlGaAs lasers operating in the 800- to 900-nm band. The remaining chapterscover direct and externalmodulation techniques,photodetectors and receiver design, and system design and applications Still, the basic elements of the present day systems are discussed: low-loss vapor-phase silica fiber and double-heterostructurelasers. Although system trials were initiated around 1979, it required several more years before a commercially attractive lightwave telecommunications system was installed in the United States. The AT&T Northeast Corridor System, operating between New York and Washington, DC, began service in January 1983, operating at a wavelength of 820 nm and a bit rate of 45 Mb/s in multimode fiber. Lightwave systems were upgraded in 1984 to 1310nm and 417 or 560 Mb/s in single-mode fiber in the United States as well as in Europe and Japan. The year 1984 also saw the Bell System broken up by the court-imposed “Modified Final Judgment” that separated the Bell operating companies into seven regional companies and left AT&T as the long distance camer as well as a telephone equipment vendor. Bell Laboratories remained with AT&T, and Bellcore was formed to serve as the R&D lab for all seven regional Bell operating companies (RBOCs). The breakup spurred a rise in diversity and competition in the communications business. The combination of technical advances in computers and communications, growing government deregulation, and apparent new business opportunities all served to raise expectations. Tremendoustechnical progresswas made during the next few years, and the choice of lightwave over copper coaxial cable or microwave relay for most longhaul transmission systems was assured. The goal of research was to improve performance, such as bitrate and repeater spacing, and to find other applications beyond point-to-point long haul telephone transmission. A completely new book, Optical Fiber Telecommunications11,was published in 1988 to summarize the lightwave R&D advancesat the time. To broaden the coverage, nonBell Laboratories authors from Bellcore (now Telcordia), Corning, Nippon Electric Corporation, and several universities were represented among the contributors. Although research results are described in OFT 11, the emphasis is much stronger on commercial applications than in the previous volume. The initial chapters of OFT 11 cover fibers, cables, and connectors, dealing with both single- and multimode fiber. Topics include vapor-phase methods for fabricating low-loss fiber operating at 13 10 and 1550 nm, understanding chromatic dispersion and nonlinear effects, and designing polarization-maintaining fiber. Another large group of chapters deals with
1.Overview
3
a range of systems for loop, intercity, interoffice, and undersea applications. A research-oriented chapter deals with coherent systems and another with possible local area network designs, including a comparison of time-division multiplexing (TDM) and wavelength division multiplexing (WDM) to efficiently utilize the fiber bandwidth. Several chapters cover practical subsystem components, such as receivers and transmitters and their reliability. Other chapters cover photonic devices, such as lasers, photodiodes, modulators, and integrated electronic and integrated optic circuits that make up the subsystems. In particular, epitaxial growth methods for InGaAsP materials suitable for 1310 and 1550nm applications and the design of high-speed single-mode lasers are discussed in these chapters. By 1995, it was clear that the time had arrived to plan for a new volume to address recent research advances and the maturing of lightwave systems. The contrast with the research and business climates of 1979 was dramatic. Sophisticatedsystem experiments were being performed utilizing the commercial and research components developed for a proven multibillion-dollar global lightwave industry. For example, 10,000-kmlengths of high-performance fiber were assembled in several laboratories around the world for nonreturn-to-zero (NRZ), soliton, and WDM transmission demonstrations. Worldwide regulatory relief stimulated the competition in both the service and hardware ends of the telecommunications business. The success in the long-haul market and the availability of relatively inexpensive components led to a wider quest for other lightwave applications in cable television and local access network markets. The development of the diode-pumped, erbium-doped fiber amplifier (EDFA) played a crucial role in enhancing the feasibility and performance of long-distance and WDM applications. By the time of publication of OFT 111 in 1997, incumbent telephone companies no longer dominated the industry. New companies were offering components and systems and other startups were providing regional, exchange, and Internet services. In 1996, AT&Tvoluntarilyseparated its long distance serviceand telephone equipment businesses to better meet the competition. The former kept the AT&T name, and the latter took on the name Lucent Technologies. Bell Labs remained with Lucent, and AT&T Labs was formed. Bellcore was put up for sale, as the consolidating and competing RBOCs found they did not need a joint lab. Because of a wealth of new information, OFTIII was divided into two books, A and B, covering systems and components, respectively. Many topics of the previous volumes, such as fibers, cables, and laser sources, are updated. But a much larger list of topics covers fields not previously included. In A , for example, transceiver design, EDFAs, laser sources, optical fiber components,planar (silica on silicon) integrated circuits, lithium niobate devices, and photonic switching are reviewed. And in By SONET (synchronous optical network) standards, fiber and cable design, fiber nonlinearities, polarization effects, solitons, terrestrial and undersea systems, high bitrate transmission, analog
4
Ivan P.Kaminow
cable systems,passive optical networks (PONS),and multiaccess networks are covered. Throughout the two books, erbium amplifiers and WDM are common themes. It is difficult to overstate the impact these two technologies have had in both generating and supporting the telecommunications revolution that coincided with their commercialintroduction. The EDFA was first reported in about 1987 by researchers at Southampton Universityin the UK and at AT&T Bell Labs. In 1990, driven by the prospect of vast savings offered by WDM transmission using EDFAs, Bell Labs began to develop long-haul WDM systems. By 1996, AT&T and Alcatel had installed the first transatlantic cable with an EDFA chain and a single 5 Gb/s optical channel. AT&T installed the first commercialterrestrial WDM system employing EDFAs in 1995. Massive deployment of WDM worldwide soon followed. WDM has made the exponential tratlic growth spurred by the coincident introduction of the Internet browser economically feasible. If increased TDM bitrates and multiple fibers were the only alternative, the enthusiastic users and investors in the Internet would have been priced out of the market.
Optical Fiber TelecommunicationsIV BA CKGROWD There was considerableexcitementin the lightwaveresearchcommunityduring the 1970s and early 1980s as wonderful new ideas emerged at a rapid pace. The monopoly telephone system providers, however, were less enthusiastic. They were accustomed to moving at their own deliberate pace, designingequipment to install in their own systems, which were expected to have a long economic life. The long-range planners projected annual telephone voice traffic growth in the United States at about 5-lo%, based on population and business growth. Recent years, on the other hand, have seen mind-numbing changes in the communication business-especially for people brought up in the telephone environment. The Internet browser spawned a tremendous growth in data traillc, which in turn encouraged visions of tremendous revenue growth. Meanwhile, advances in WDM technology and its wide deploymentsynergistically supported the Internet traffic and enthusiasm. As a result, entrepreneurs invested billions of dollars in many companies vying for the same slice of pie. The frenzy reached a peak in the spring of 2000 and then rapidly melted down as investors realized that the increased network capacity had already outstripped demand. As of October 2001, the lightwave community is waiting for a recovery from the current industry collapse. Nevertheless, the technical advances achieved during these last five years will continue to impact telecommunications for years to come. Thus, we are proud to present a comprehensive and forward-lookingaccount of these accomplishments.
1.Overview
5
Survey of O F T W A and B Advances in optical network architectures have followed component innovations. For example, the low loss fiber and double heterostructure laser enabled the first lightwave system generation; and the EDFA has enabled the WDM generation. Novel components (such as tunable lasers, MEMS switches, and planar waveguide devices) are making possible more sophisticatedoptical networks. At the same time, practical network implementationsuncover the need for added device functionality and very low cost points. For example, 40 Gb/s systems need dynamic dispersion and PMD compensationto overcome system impairments. We have divided OFTIV into two books: book A comprises the component chapters and book B the system and system impairment chapters.
BOOKA: COMPONENTS
Design of Optical Fibers for Communications Systems (Chapter 2) Optical fiber has been a key element in each of the previous volumes of OFT. The present chapter by DiGiovanni, Boncek, Golowich, Das, Blyler, and White reflects a maturation of the field: fiber performance must now go beyond simple low attenuation and must exhibit critical characteristicsto support the high speeds and long routes on terrestrial and undersea systems. At the same time, fiber for the metropolitan and access markets must meet demandingprice points. The chapter reviews the design criteria for a variety of fibers of current commercial interest. For the traditional long-haul market, impairments such as dispersion slope and polarization mode dispersion (PMD) that were negligible in earlier systems are now limiting factors. If improved fiber design is unable to overcome these limits, new components will be required to solve the problem. These issues are addressed again from different points of view in later systems and components chapters in O F T N A and B. The present chapter also reviews a variety of new low-cost fiber designs for emerging metropolitan and access markets. Further down the network chain, the design of multimode glass and plastic fiber for the highly cost-sensitivelocal area network market are also explored. Finally, current research on hollow core and photonic bandgap fiber structures is summarized.
New Materials for Optical Amplifiers (Chapter 3)
In addition to transport, fiber plays an important role as an amplifying medium. Aluminum-doped silica has been the only important commercial host and erbium the major amplifying dopant. Happily, erbium is soluble in AI-silica and provides gain at the attenuation minimum for silica transmission fiber. Still, researchers are exploring other means for satisfying demands
6
Ivan P. Kaminow
for wider bandwidth in the 1550nm region as well as in bands that might be supported by other rare-earth ions, which have low efficiency in silica hosts. Ellison and Minelly review research on new fiber materials, including fluorides, alumina-doped silica, antimony silicates, and tellurite. They also report on extended band erbium-doped fiber amplifiers (EDFAs), thulium-doped fiber amplifiers, and 980 nm ytterbium fiber lasers for pumping EDFAs.
Advances in Erbium-Doped Fiber Amplifiers (Chapter 4) The development of practical EDFAs has ushered in a generation of dense WDM (DWDM) optical networks. These systems go beyond single frequency or even multifrequency point-to-point links to dynamic networks that can be reconfigured by adddrop multiplexers or optical cross-connects to meet varying system demands. Such networks place new requirements on the EDFAs: they must maintain flatness over many links, and they must recover from sudden drops or adds of channels. And economics drives designs that provide more channels and denser spacing of channels. Srivastava and Sun summarize recent advances in EDFA design and means for coping with the challenges mentioned above. In particular, they treat long wave L-band amplifiers, which have more than doubled the conventional C-band to 84nm. They also treat combinations of EDFA and Raman amplification, and dynamic control of gain flatness.
Raman Amplification in Lightwave Communication Systems (Chapter 5) Raman amplification in fibers has been an intellectual curiosity for nearly 30 years; the large pump powers and long lengths required made Raman amplifiers seem impractical. The advent of the EDFA appeared to drive a stake into the heart of Raman amplXers.Now, however, Raman amplifiers are rising along with the needs of submarine and ultralong-haul systems. More powerful practical diode pumps have become available; and the ability to provide gain at any wavelength and with low effective noise figure is now recognized as essential for these systems. Rottwitt and Stentz review the advances in distributed and lumped Raman amplifierswith emphasis on noise performance and recent system experiments.
Electrooptic Modulators (Chapter 6) Modulators put the payload on the optical carrier and have been a focus of attention from the beginning. Direct modulation of the laser current is often the cheapest solution where laser linewidth and chirp are not important. However, for high performance systems, external modulators are needed. Modulators based on the electrooptic effect have proven most versatile in meeting performance requirements, although cost may be a constraint.
1.Overview
7
Titanium-diffusedlithium niobate has been the natural choice of material, in that no commercial substitutes have emerged in nearly 30 years. However, integrated semiconductor electroabsorption modulators are now offering strong competition on the cost and performance fronts. Mahapatra and Murphy briefly compare electroabsorption-modulated lasers (EMLs) and electrooptic modulators. They then focus on titaniumdiffused lithium niobate modulators for lightwave systems. They cover fabrication methods, component design, system requirements, and modulator performance. Mach-Zehnder modulators are capable of speeds in excess of 40Gb/s and have the ability to control chirp from positive through zero to negative values for various system requirements. Finally, the authors survey research on polymer electroopticmodulators, which offer the prospect of lower cost and novel uses.
Optical Switching in Transport Networks: Applications, Requirements, Architectures, Technologies, and Solutions (Chapter 7) Early DWDM optical line systems provided simple point-to-point links between electronic end terminals without allowing access to the intermediate wavelength channels. Today’s systems carry over 100 channels per fiber and new technologies allow intermediate routing of wavelengths at add/drop multiplexers and optical cross-connects. These new capabilities allow “optical layer networking,” an architecture with great flexibility and intelligence. AI-Salameh, Korotky, Levy, Murphy, Patel, Richards, and Tentarelli explore the use of optical switchingin modern networking architectures. After reviewing principles of networking, they consider in detail various aspects of the topic. The performance and requirements for an optical cross connect (OXC) for opaque (with an electronic interface and/or electronic switch fabric) and transparent (all-optical) technologies are compared. Also, the applications of the OXC in areas such as provisioning, protection, and restoration are reviewed. Note that an OXC has all-optical ports but may have internal electronics at the interfaces and switch fabric. Finally, several demonstration OXCs are studied, including small optical switch fabrics, wavelength-selective OXCs, and large strictly nonblocking cross connects employing microelectromechanical system (MEMS) technology. These switches are expected to be needed soon at core network nodes with 1000 x 1000 ports.
Applications for Optical Switch Fabrics (Chapter 8) Whereas the previous chapter looked at OXCs from the point of view of the network designer, Zirngibl focuses on the physical design of OXCs with capacities greater than 1Tb/s. He considers various design options including MEMS switch fabrics, transparent and opaque variants, and nonwavelength-blocking
8
Ivan P.Kaminow
configurations He finds that transport in the backplane for very large capacity (bitrate x port number) requires optics in the interconnectsand switch fabric. He goes beyond the cross-connect application, which is a slowly reconfigurable circuit switch, to consider the possibility of a high-capacity packet switch, which, although schematically similar to an OXC, must switch in times short relative to a packet length. Again the backplaneproblem dictatesan optical fabric and interconnects. He proposes tunable lasers in conjunction with a waveguide grating router as the fast optical switch fabric.
Planar Lightwave Devices for WDM (Chapter 9) The notion of integrated optical circuits, in analogy with integrated electronic circuits, has been in the air for over 30 years, but the vision of large-scale integration has never materialized. Nevertheless, the concept of small-scale planar waveguide circuits has paid off handsomely. Optical waveguiding provides efficientinteractions in lasers and modulators, and novel functionalityin waveguide grating routers and Bragg gratings. These elements are often linked together with waveguides. Doerr updates recent progress in the design of planar waveguides, starting with waveguide propagation analysis and the design of the star coupler and waveguide grating router (or arrayed waveguide grating). He goes on to describe a large number of innovativeplanar devices such as the dynamic gain equalizer, wavelength selective cross connect, wavelength adddrop, dynamic dispersion compensator, and the multifrequency laser. Finally, he compares various waveguide materials: silica, lithium niobate, semiconductor, and polymer.
Fiber Grating Devices in High-Performance Optical Communication Systems (Chapter 10) The fiber Bragg grating is ideally suited to lightwave systems because of the ease of integrating it into the fiber structure. The technology for economically fabricating gratings has developed over a relatively short period, and these devices have found a number of applicationsto which they are uniquely suited. For example, they are used to stabilize lasers, to provide gain flattening in EDFAs, and to separate closely spaced WDM channels in adddrops. Strasser and Erdogan review the materials aspects of the major approaches to fiber grating fabrication. Then they treat the properties of fiber gratings analytically. Finally, they review the device properties and applications of fiber gratings.
Pump Laser Diodes (Chapter 11) Although EDFAs were known as early as 1986, it was not until a high-power 1480nm semiconductorpump laser was demonstratedthat people took notice.
1.Overview
9
Earlier, expensive and bulky argon ion lasers provided the pump power. Later, 980nm pump lasers were shown to be effective. Recent interest in Raman amplifiers has also generated a new interest in 1400nm pumps. Ironically, the first 1480nm pump diode that gave life to EDFAs was developed for a Raman amplifier application. Schmidt, Mohrdiek, and Harder review the design and performance of 980 and 1480nm pump lasers. They go on to compare devices at the two wavelengths, and discuss pump reliability and diode packaging.
TelecommunicationLasers (Chapter 12) Semiconductor diode lasers have undergone years of refinement to satisfy the demands of a wide range of telecommunication systems. Long-haul terrestrial and undersea systems demand reliability, speed, and low chirp; short-reach systems demand low cost; and analog cable TV systems demand high power and linearity. Ackerman, Eng, Johnson, Ketelsen, Kiely, and Mason survey the design and performance of these and other lasers. They also discuss electroabsorption modulated lasers (EMLs) at speeds up to 40 Gb/s and a wide variety of tunable lasers.
VCSELs for Metro Communications (Chapter 13) Vertical cavity surface emitting lasers (VCSELs) are employed as low-cost sources in local area networks at 850nm. Their cost advantage stems from the ease of coupling to fiber and the ability to do wafer-scale testing to eliminate bad devices. Recent advances have permitted the design of efficient long wavelength diodes in the 1300-1600 nm range. Chang-Hasnain describes the design of VCSELs in the 1310 and 1550nm bands for application in the metropolitan market, where cost is key. She also describes tunable designs that promise to reduce the cost of sparing lasers.
Semiconductor Optical Amplifers (Chapter 14) The semiconductor gain element has been known from the beginning, but it was fraught with difficulties as a practical transmission line amplifier: it was difficult to reduce reflections, and its short time constant led to unacceptable nonlinear effects. The advent of the EDFA practically wiped out interest in the semiconductor optical amplifier (SOA) as a gain element. However, new applications based on its fast response time have revived interest in SOAs. Spiekman reviews recent work to overcome the limitations on SOAs for amplification in single-frequency and WDM systems. The applications of main interest, however, are in optical signal processing, where SOAs are used in wavelength conversion, optical time division multiplexing, optical phase
10
Ivan P. Kaminow
conjugation, and all-optical regeneration. The latter topic is covered in detail in the following chapter.
All-Optical Regeneration: Principles and WDM Implementation (Chapter 15) A basic component in long-haul lightwave systems is the electronic regenerator. It has three functions: reamplifying, reshaping, and retiming the optical pulses. The EDFA is a 1R regenerator; regenerators without retiming are 2R; but a full-scale repeater is a 3R regenerator. A separate 3R electronic regenerator is required for each WDM channel after a fixed system span. As the bitrate increases, these regenerators become more expensive and physically more difficult to realize. The goal of ultralong-haul systems is to eliminate or minimize the need for electronic regenerators (see Chapter 5 in Volume B). Leclerc, Lavigne, Chiaroni, and Desurvire describe another approach, the all-optical3R regenerator. They describe avariety of techniques that have been demonstratedfor both single channel and WDM regenerators. They argue that at some bitrates, say 40 Gb/s, the optical and electronic alternatives may be equally difficult and expensive to realize, but at higher rates the all-optical version may dominate.
High Bitrate Tkansmitters, Receivers, and Electronics (Chapter 16) In high-speed lightwave systems, the optical components usually steal the spotlight. However, the high bitrate electronics in the terminals are often the limiting components. Kasper, Mizuhara, and Chen review the design of practical high bitrate (10 and 40 Gb/s) receivers, transmitters, and electronic circuits in three separate sections. The first section reviews the performance of various detectors, analyzes receiver sensitivity, and considers system impairments. The second section covers directly and externally modulated transmitters and modulation formats like return-to-zero (RZ) and chirped RZ (CRZ). The final section covers the electronic circuit elements found in the transmitters and receivers, including broadband amplifiers, clock and data recovery circuits, and multiplexers.
BOOK B: SYSTEMS AND IMPAIRMENTS Growth of the Internet (Chapter 2) The explosion in the telecommunicationsmarketplace is usually attributed to the exponentialgrowth of the Internet, which began its rise with the introduction of the Netscape browser in 1996. Voice traffic continues to grow steadily, but data traffic is said to have already matched or overtaken it. A lot of selfserving myth and hyperbole surround these fuzzy statistics. Certainly claims of doubling data t r a c every three months helped to sustain the market frenzy.
1.Overview
11
On the other hand, the fact that revenues from voice traffic still far exceed revenues from data was not widely circulated. Coffman and Odlyzko have been studying the actual growth of Internet traffic for several years by gathering quantitative data from service providers and other reliable sources. The availability of data has been shrinking as the Internet has become more commercial and fragmented. Still, they find that, while there may have been early bursts of three-month doubling, the overall sustained rate is an annual doubling. An annual doubling is a very powerful growth rate; and, if it continues, it will not be long before the demand catches up with the network capacity. Yet, with prices dropping at a comparable rate, faster traffic growth may be required for strong revenue growth.
Optical Network Architecture Evolution (Chapter 3) The telephone network architecture has evolved over more than a century to provide highly reliable voice connections to a global network of hundreds of millions of telephones served by different providers. Data networks, on the other hand, have developed in a more ad hoc fashion with the goal of connecting a few terminals with a range of needs at the lowest price in the shortest time. Reliability, while important, is not the prime concern. Strand gives a tutorial review of the Optical Transport Network employed by telephone service providers for intercity applications. He discussesthe techniques used to satisfy the traditional requirements for reliability, restoration, and interoperability.He includes a refresher on SONET (SDH). He discusses architectural changes brought on by optical fiber in the physical layer and the use of optical layer cross connects. Topics include all-optical domains, protection switching, rings, the transport control plane, and business trends.
Undersea Communication Systems (Chapter 4) The oceans provide a unique environment for long-haul communication systems. Unlike terrestrial systems, each design starts with a clean slate; there are no legacy cables, repeater huts, or rights-of-way in place and few international standards to limit the design. Moreover, there are extreme economic constraints and technological challenges For these reasons, submarine systems designers have been the first to risk adopting new and untried technologies, leading the way for the terrestrial ultralong-haul systemdesigners (see Chapter 5). Following a brief historical introduction, Bergano gives a tutorial review of some of the technologies that promise to enable capacities of 2Tbh on a single fiber over transoceanic spans. The technologies include the chirped RZ (CRZ) modulation format, which is compared briefly with NRZ, RZ, and dispersion-managed solitons (see Chapters 5,6, and 7 for more on this topic). He also discusses measures of system performance (the Q-factor), forward
12
Ivan P.Kaminow
error correcting(FEC) codes (see Chapters 5 and 17), long-haulsystem design, and future trends.
High Capacity, Ultralong-Haul Transmission (Chapter 5) The major hardware expense for long-haul terrestrial systems is in electronic terminals, repeaters, and line cards. Since WDM systems permit traffic with various destinationsto be bundled on individualwavelengths, great savingscan be realized if the unrepeatered reach can be extended to 2000-5000 km, allowing traffic to pass through nodes without optical-to-electrical (O/E) conversion. As noted in connection with Chapter 4, some of the technologypioneered in undersea systems can be adapted in terrestrial systems but with the added complexities of legacy systems and standards. On the other hand, the terrestrial systems can add the flexibilityof optical networkingby employingoptical routing in add/drops and OXCs (see Chapters 7 and 8) at intermediatepoints. Zyskind, Barry, Pendock, and Cahill review the technologies needed to design ultralong-haul (ULH) systems. The technologies include EDFAs and distributed Raman amplification, novel modulation formats, FEC, and gain flattening. They also treat transmission impairments (see later chapters in this book) such as the characteristics of fibers and compensators needed to deal with chromatic dispersion and PMD. Finally, they discuss the advantages of optical networking in the efficient distribution of data using IP (Internet Protocol) directly on wavelengths with meshes rather than SONET rings.
Pseudo-Linear Transmission of High-speed TDM Signals: 40 and 160 Gb/s (Chapter 6) A reduction in the cost and complexity of electronic and optoelectroniccomponents can be realized by an increase in channel bitrate, as well as by the ULH techniquesmentioned in Chapter 5. The higher bitrates, 40 and 160Gb/s, present their own challenges, among them the fact that the required energy per bit leads to power levels that produce nonlinear pulse distortions. Newly discovered techniques of pseudo-linear transmission offer a means for dealing with the problem. They involve a complex optimization of modulation format, dispersion mapping, and nonlinearity. Pseudo-linear transmission occupies a space somewhere between dispersion-mapped linear transmission and nonlinear soliton transmission (see Chapter 7). Essiambre, Raybon, and Mikkelsen first present an extensive analysis of pseudo-linear transmission and then review TDM transmission experiments at 40 and 160Gb/s.
Dispersion Managed Solitons and Chirped RZ: What Is the Difference? (Chapter 7) Menyuk, Carter, Kath, and Mu trace the evolution of soliton transmission to its present incarnation as Dispersion Managed Soliton (DMS) transmission
1.Overview
13
and the evolution of NRZ transmission to its present incarnation as CRZ transmission. Both approaches depend on an optimization of modulation format, dispersion mapping, and nonlinearity, defined as pseudo-linear transmission in Chapter 6 and here as “quasi-linear” transmission. The authors show how both DMS and CRZ exhibit aspects of linear transmission despite their dependence on the nonlinear Icerr effect. Remarkably, they argue that, despite widely disparate starting points and independent reasoning, the two approaches unwittingly converge in the same place. Still, on their way to convergence, DMS and CRZ pulses exhibit different characteristicsthat suit them to different applications: For example, CRZ produces pulses that merge in transit along a wide undersea span and reform only at the receiver ashore, while DMS produces pulses that reform periodically, thereby permitting access at intermediate adddrops.
Metropolitan Optical Networks (Chapter 8) For many years the long-haul domain has been the happy hunting ground for lightwave systems, since the cost of expensive hardware can be shared among many users. Now that component costs are moderating, the focus is on the metropolitan domain where costs cannot be spread as widely. Metropolitan regions generally span ranges of 10 to 100km and provide the interface with access networks (see Chapters 9, 10, and 11). SONETBDH rings, installed to serve voice traffic, dominate metropolitan networks today. Ghani, Pan, and Chen trace the developing access users, such as Internet service providers, local area networks, and storage area networks. They discuss a number of WDM metropolitan applications to better serve them, based on optical networking via optical rings, optical adddrops, and OXCs. They also consider 1P over wavelengths to replace SONET. Finally, they discuss possible economical migration paths from the present architecture to the optical metropolitan networks.
The Evolution of Cable TV Networks (Chapter 9) Coaxial analog cable TV networks were substantially upgraded in the 1990s by the introduction of linear lasers and single-modefiber. Hybrid Fiber Coax (HFC) systems were able to deliver in excess of 80 channels of analog video plus a wide band suitable for digital broadcast and interactive services over a distance of 60 km. Currently high-speed Internet access and voice-over-IP telephony have become available,making HFC part of the telecommunications access network. Lu and Sniezko outline past, present, and future HFC architectures. In particular, the mini fiber node (mFN) architecture provides added capacity for two-way digital as well as analog broadcast services. They consider a number of mFNvariants based on advancesin RF, lightwave,and DSP (digital signal processor) technologies that promise to provide better performance at lower cost.
14
Ivan P.Kaminow
Optical Access Networks (Chapter 10) The access portion of the telephone network, connecting the central office to the residence, is called the “loop.” By 1990 half the new loops in the United Stateswere served by digital loop carrier (DLC), a fiber severalmiles long from the central office to aremote terminal in aneighborhoodthat connects to about 100 homes with analog signals over twisted pairs. Despite much anticipation, fiber hasn’t gotten much closer to residences since. The reason is that none of the approachesproposed so far is competitivewith existing technology for the applicationspeople will buy. Harstead and van Heyningen survey numerous proposals for Fiber-in-theLoop (FITL) and Fiber-to-the-X (FTTX), where X = Curb, Home, Desktop, etc. They consider the applications and costs of these systems. Considerable creativity and thought have been devoted to fiber in the access network, but the economics still do not work because the costs cannot be divided among a suflicient number of users. An access technology that is successful is Digital Subscriber Line (DSL) for providing high-speed Internet over twisted pairs in the loop. DSL is reviewed in an Appendix.
Beyond Gigabit: Development and Application of High-speed Ethernet Technology (Chapter 11) Ethernet is a simple protocol for sharing a local area network (LAN). Most of the data on the Internet start as Ethernet packets generated by desktop computers and system servers. Because of their ubiquity, Ethernet line cards are cheap and easy to install. Many people now see Ethernet as the universal protocol for optical packet networks. Its speed has already increased to 1000Mb/s, and 10 Gb/s is on the way. Lam describes the Ethernet system in detail from protocols to hardware, including 10 Gb/s Ethernet. He shows applications in LANs, campus, metropolitan, and long distance networks.
Photonic Simulation Tools (Chapter 12) In the old days, new devices or systems were sketched on a pad, a prototype was put together in the lab, and its performance tested. In the present climate, physical complexity and the expense and time required rule out this bruteforce approach, at least in the early design phase. Instead, individual groups have developed their own computer simulators to test numerous variations in a short time with little laboratory expense. Now, several commercial vendors offer general-purposesimulators for optical device and system development. Lowery relates the history of lightwave simulators and explains how they work and what they can do. The user operates from a graphic user interface (GUI) to select elements from a library and combine them. The simulated device or system can then be run and measured as in the lab to determine
1.Overview
15
attributes like the eye-diagram or bit-error-rate. In the end, a physical prototype is required because of limits on computation speed among other reasons.
THE PRECEDING CRAPTERS RAVE DEALT WITH SYSTEM DESIGN; THE REMAIMNG CHAPTERS DEAL WlTH SYSTEM IMPAIRMENTS AND METHODS FOR MITIGATING THEM Nonlinear Optical Effects in WDM Systems (Chapter 13) Nonlinear effects have been mentioned in different contexts in several of the earlier system chapters. The Kerr effect is an intrinsic property of glass that causes a change in refractive index proportional to the optical power. Bayvel and Killey give a comprehensive review of intensity-dependent behavior based on the Ken effect. They cover such topics as self-phase modulation, cross-phase modulation, four-wave mixing, and distortions in NRZ and RZ systems.
F%ied and "unable Management of Fiber Chromatic Dispersion (Chapter 14) Chromatic dispersion is a linear effect and as such can be compensated by adding the complementary dispersion before any significant nonlinearities intervene. Nonlinearities do intervene in many of the systems previously discussed so that periodic dispersion mapping is required to manage them. Willner and Hoanca present a thorough taxonomy of techniques for compensating dispersion in transmission fiber. They cover fixed compensation by fibers and gratings, as well as tunable compensation by gratings and other novel devices. They also catalog the reasons for incorporating dynamic as well as k e d compensation in systems.
Polarization Mode Dispersion (Chapter 15) Polarization mode dispersion (PMD), like chromatic dispersion, is a linear effect that can be compensated in principle. However, fluctuationsin the polarization mode and fiber birefringence produced by the environment lead to a dispersion that varies statistically with time and frequency. The statistical nature makes PMD difficult to measure and compensate for. Nevertheless, it is an impairment that can kill a system, particularly when the bitrate is large (> 10 Gb/s) or the fiber has poor PMD performance. Nelson, Jopson, and Kogelnik offer an exhaustive survey of PMD covering the basic concepts, measurement techniques, PMD measurement, PMD statistics for first- and higher orders, PMD simulation and emulation, system impairments, and mitigation methods. Both optical and electrical PMD compensation (see Chapter 18) are considered.
16
Ivan P. Kamhow
Bandwidth Efficient Formats for Digital Fiber Transmission Systems (Chapter 16) Early lightwave systems employedNRZ modulation; newer long-haul systems are using RZ and chirped RZ to obtain better performance. One goal of system designersis to increase spectral efficiencyby reducing the RF spectrum required to transmit a given bitrate. Conradi examines a number of modulation formats well known to radio engineers to see if lightwave systems might benefit from their application. He reviews the theory and DWDM experiments for such formats as M-ary ASK, duo-binary, and optical single-sideband.He also examines RZ formats combined with various types of phase modulation, some of which are related to discussions of CRZ in the previous Chapters 4-7.
Error-Control Coding Techniques and Applications (Chapter 17) Error-correctingcodes axe widelyused in electronics, e.g., in compactdisc players, to radically improve system performance at modest cost. Similar forward error correcting codes (FEC) are used in undersea systems (see Chapter 4) and are planned for ULH systems (Chapter 5). Win, Georghiades, Kumar, and Lu give a tutorial introduction to coding theory and discuss its application to lightwave systems. They conclude with a critical survey of recent literature on FEC applications in lightwave systems, where FEC provides substantial system gains.
Equalization Techniques for Mitigating Transmission Impairments (Chapter 18) Chapters 14and 15describe optical means for compensatingthe linear impairments caused by chromatic dispersion and PMD. Chapters 16 and 17 describe two electronic means for reducing errors by novel modulation formats and by FEC. This chapter discusses a third electronic means for improvingperformance using equalizer circuits in the receiving terminal, which in principle can be added to upgrade an existing system. Equalization is widely used in telephony and other electronic applications. It is now on the verge of application in lightwave systems. Win, Vitetta, and Winters point out the challengesencounteredin lightwave applications and survey the mathematical techniques that can be employed to mitigate many of the impairments mentioned in previous chapters. They also describe some of the recent experimental implementations of equalizers. Additional discussion of PMD equalizers can be found in Chapter 15.
Duty Cycle lOO%(Ng)
33%
50%
20%
10%
40 Gbls 2
Single channel Distance of 80 km
TrueWave@ with D = 4 ps/(km nm)
I
-250
-125
0
Pceromp Ipdnmm)
-250
-125
0
-250
-125
0
-250
k r o m p (pshml
k c a m p (p4nml
-125
-250
0
-2-1 0 I 2 3 Eye Closure Penalty (dB) -12J
0
k c o m p lpdnrnl
Prccomp lpdnrni
Plate 1 Eye closure penalties as a function of modulation formats and launch (average) power for 40-Gbh single-channel transmission over 80 km of TrueWaveTMfiber [TrueWaveTMparameters are given in Table 6.1 except for the value of D = 4ps/(km nm) here]. Full dispersion and dispersion slope compensation are assumed before the postcompensation. Each plot in the matrix of plots shows the color-coded eye closure penalties Ceyeas a function of pre- and postcompensation. Only a small improvement in eye opening (essentially the back-to-back difference in eye opening) can be seen when decreasing the duty cycle. Only when the duty cycle is reduced to a value as low as 10% is a significant improvement in transmission observed. Amplifier noise is not included.
Duty Cycle
---
33%
I
5
RFrornp lpdnm)
F-omp lpdnml
20%
I
Rsomp
lpdnml
10% I
F-omp (~Jn,nm)
R-mp
Ipdoml
Plate 2 Identical to Plate 1 except for STD unshifted fiber (parameters given in Table 6.1). For large duty cycles (NRZ and 50% duty cycle), transmission is limited to lower powers than TrueWaveTMfiber but rapidly increases as the duty cycle decreases.
40 Gb/s Single channel
12 dBm 8 spans of 80 !un
TrueWaveTM/DSF with
D = 2 ps/(km nm)
- I O
1
2
3
4
Eye Closure Penalty (dB)
Recomp. (pdnm)
Reeomp. (pdnm)
Recomp. (pdnm)
Recomp. (pdnm)
Plate 3 Eye closure penalties after 8 spans of 80 km as a function of residual dispersion per span and modulation format. The transmission fiber has the same parameters as TrueWaveTM (Table 6.1) except we use here D = 2 ps/(km nm). The dashed lines are the points of zero net residual dispersion. Two regimes of transmission are present. A first regime (solitonic regime) has optimum transmission with a net positive residual dispersion. In this regime the solitonic effect is at play and is responsiblefor the compensationof dispersionby nonlinearity (even for NRZ!). The second regime (pseudo-linearregime) has its optimum transmission at zero net residualdispersion.
Residual DisDersion per Span 0
40 Gb/s Single channel
12 dBm 8 spans of 80 km
TrueWavem with
I
D = 4 pd&m nm)
-
1
0
1
2
3
4
Eye Closure Penalty (dB) -la,
-200 -200
-100
0
Reeomp. (pdnm)
a
-100
0
Precomp. (pdnm)
-200 -100 0 Reeomp (plnm)
Precomp. (pdnm)
Plate 4 Same as Plate 3 except D = 4ps/(km nm).
Residual Dispersion per Span 8ps/m 16 ps/nm 24 pdnm
0
_____________ -_____________
40 Gb/s Single channel
12 dBm 8 spans of 80 km
TrueWavem with
D = 8 pd&m nm)
M,. .. .. , -1
0
1
2
3
1
Eye Closure Penalty (dB)
m
-100
o
Recomp. (jdnm)
-200
-100
o
PReomp (pdnm)
-200
-100
o
Reeomp. (pslnm)
Plate 5 Same as Plate 3 except D = 8 ps/(km nm).
Residual Dispersion per Span
40 Gb/s Single channel
12 dBm 8 spans of 80 km
STD Unshifted Fiber with D = 17 pd@m nm)
-
1
0
1
2
3
4
Eye Closure Penalty (dB)
Rsomp. (pS/nrnl
Rsomp. (pslnml
F’recomp. ( g n m l
Pr~comp.(pdnm)
Plate 6 Same as Plate 3 except for STD unshifted fiber D = 17ps/(km nm) and A,ff = 80 km2.
Fiber 1 35
Fiber 2
I
.-2 2 5 ‘ g 20 F
15 10
15,J
1520
1530 1540 1550 Wavelength [nm]
1560
1510
1520
1530 1540 1550 Wavelength [nm]
1560
Plate 7 Contour plots of the simultaneous DGD measurements of two fibers in the same embedded cable over a 36-day period. The mean DGDs averaged over time and wavelength were 2.75 and 2.89 ps for fibers 1 and 2, respectively. Data is courtesy of Magnus Karlsson.
Chapter 2
Growth of the Internet
Kerry G. Coffman AT&T Labs-Research, Middletown, New Jersey
Andrew M. Odlyzko University of Minnesota, Minneapolis, Minnesota
Abstract The Internet is the main cause of the recent explosion of activity in opticalfiber telecommunications.The high growth rates observed on the Internet, and the popular perception that growth rates were even higher, led to an upsurge in research, development, and investment in telecommunications.The telecom crash of 2000 occurred when investors realized that transmission capacity in place and under construction greatly exceeded actual traffic demand. This chapter discussesthe growth of the Internet and compares it with that of other communication services. It also presents speculations about future developments. Internet traffic is growing, approximatelydoublingeach year. There are reasonable arguments that it will continue to grow at this rate for the rest of this decade. If this happens, then in a few years we may have a rough balance between supply and demand.
1. Introduction Optical fiber communication was initially developed for the voice phone system. The feverish level of activity that we have experienced since the late 199Os, though, was caused primarily by the rapidly rising demand for Internet connectivity. The Internet has been growing at unprecedented rates. Moreover, because it is versatile and penetrates deeply into the economy, it is affecting all of society, and therefore has attracted inordinate amounts of public attention. The aim of this chapter is to summarize the current state of knowledge about the growth rates of the Internet, with special attention paid to the implications for fiber optic transmission. We also attempt to put the growth rates of the Internet into the proper context by providing comparisons with other communications services. The overwhelmingly predominant view has been that Internet traffic (as measured in bytes received by customers) doubles every 3 or 4 months. 17 OPTICAL FIBER TELECOMMUNICATIONS, VOLUME TVB
Copyright 0 2002, Elsevier Scienm (USA).
ALI rightsof reproduction in any form reserved. ISBN &12-395173-9
18
Kerry G. Coffman and Andrew M. Odlyzko
Such unprecedented rates (corresponding to traffic increasing by factors of between 8 and 16 each year) did prevail within the United States during the crucial 2-year period of 1995 and 1996, when the Internet first burst onto the scene as a major new factor with the potential to transform the economy. However, as we pointed out in [CoffmanOl] (written in early 1998, based on data through the end of 1997), by 1997those growth rates subsided to approximate the doubling of traffic each year that had been experienced in the early 1990s. A more recent study [CoffmanO2] provided much more evidence, and in particular more recent evidence, that traffic has about doubled each year since 1997. (We use a doubling of traffic each year to refer to growth rates between 70 and 150% per year, with the wide range reflecting the uncertainties in the estimates.) Other recent observers also found that Internet traffic is about doubling each year. The evidence was always plentiful, and the only thing lacking was the interest in investigatingthe question. By 2000, though, the myth of Internet traffic doubling every 3 or 4 months was getting hard to accept. Very simple arithmetic shows that such growth rates, had they been sustained throughout the period from 1995 (when they did hold) to the end of 2000, would have produced absurdly high tr&c volumes. For example, at the end of 1994, traffic on the NSFNet backbone, which was well instrumented, came to about 15TB/month. Had just that traffic grown at 1300% per year (which is what a doubling every 3 months corresponds to), by the end of 2000, there would have been about 250,000,000 TB/month of backbone traffic in the United States. If we assume there are 150million Internet users in the United States, that would produce a data flow of about 5Mb/s for each user around the clock. The assumption of a doubling of traffic every 4 months produces traffic volumes that are only slightly less absurd. Table 2.1 shows our estimates for traffic on the Internet. The data for 1990 through 1994 is that for the NSFNet backbone, and therefore is very precise. It is incomplete only to the extent of neglecting what is thought to have been small fractions of traffic that went completely through other backbones. The data for 1996 through 2000 are our estimates, and the wide ranges reflect the uncertainties caused by the lack of comprehensive data. Table 2.2 presents our estimates of the tr&c on various long-distance networks at the end of 2000. The voice network still dominated, but it will likely be surpassed by the public Internet within a year or two. (For details of the measurements used to convert voice traffic to terabytes and related issues, see [CoffmanOl].) In terms of bandwidth, the Internet is already dominant. However, it is hard to obtain good figures, since, as we discuss later, the bandwidth of Internet backbones jumps erratically. In terms of dollars, though, voice still provides the lion’s share (well over 80%) of total revenues. We concentrate in this chapter (as in our previous papers [CoffmanOl, CoffmanO21) on the growth rates in Internet tr&c, as measured in bytes. For many purposes, it is the other measures, namely bandwidth and revenues, that are more important.
2. Growth of the Internet
19
Table 2.1 Traffic on Internet Backbones in the United States. Data are estimated traffic in terabytes (TB) during December of that year
Year
TB/month
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000
1.o 2.0 4.4 8.3 16.3 ? 1,500 2,5004,000 5,00&8,000 10,00&16,000 20,00&35,000
Tuble 2.2 Traffic on U.S. Long-Distance Networks, Year-End 2000
U.S. voice
Internet Other public data networks Private line
53,000 20,00&35,000 3,000 6,000-1 1,000
The reason we look at traffic is that we find more regularity there, and in the long run, we expect that there will be direct (although not linear) relations between traffic and the other measures. In particular, based on what we have observed so far, we expect capacity to grow somewhat faster than traffic. The studies of [CoffmanOl, Coffman021 led to the proposal of a new form of Moore’s Law, namely that a doubling of Internet tr&c each year is a natural growth rate. This hypothesis is supported by the estimates of Table 2.1, as well as by evidence presented in [CoffmanOl, CoffmanO2] of many institutions whose data traffic has been growing at about that rate for many years. This “law” is discussed further in Section 8. It is not a law of nature, but rather, like the Moore’s Law for semiconductors,a reflection of the complicated interactions of technology, economics, and sociology. Whether this “law” continues to hold or not will have important implications for the fiberoptic transmission industry.
20
Kerry G. Coffman and Andrew M. Odlyzko
Much of this chapter, especially Sections6 8 , is based on our earlier studies [CoffmanOl,CoffmanO2]. In Section 2, we present yet more evidence of how often popular perception and subsequent technologyand investmentdecisions are colored by myths that are easy to disprove, but which nobody had bothered to disprove for an astonishingly long time. In Section 3, we look at historical growth rates of various communication services and how they compare to the much higher growth rate of the Internet. Section4 is a brief review of the history of the Internet. Section 5 discusses some of the various types of growth rates that are relevant in different contexts. Section 6 presents the evidence about Internet traffic growth rates we have been able to assemble. Section7 is devoted to new sources of traffic that might create sudden surges of demand, such as Napster. Section 8 discusses the conventional Moore’s Law and the analog we are proposing for data traffic. Section 9 suggests a way of thinking about data-traffic growth, based on an analogy with the computer industry. Finally, Section 10 presents our conclusions.
2. Growth Myths and Reality Internet growth is an unusual subject, in that it has been attracting enormous attention but very little serious study. In particular, the general consensus has been that Internet traffic is doubling every 3 or 4 months. Yet no real evidence of that astronomical rate of growth was ever presented. As we discuss later, Internet traffic did grow at such rates in 1995 and 1996, but before and since it has been about doubling each year. At this point, we would like to point out the need for careful quantitative data in evaluating any claims about growth rates. Some examples of public claims that do not match reality are presented in [Coffman02].Here we discuss another case, this one concerning the widely held belief that any capacity that is installed will be quickly saturated. The British JANET network, which provides connectivity to British academic and research institutions, will be discussed in more detail later. What is important is that it is large (with three OC3 links across the Atlantic at the end of 2000), and has traffic statisticsgoing back several years available at http://bill.ja.net/. A press release, available at http://~.ja.net/press_release/archive~nnounce/index.html as “Increase in Transatlantic Bandwidth-28 May 1998” (but actually dated 3 June 1998), describedwhat happened when JANET’Stransatlanticlink was increasedfrom a single T3 to two T3s: With effect from Thursday 28 May 1998, JANET has been running a second T3 (45 Mbit/s) link to the North American Internet, bringing the total transatlantic bandwidth available to JANET to 90 Mbit/s. . . . Usage of the new capacity has been brisk, with the afternoon usage levels reaching in excess of 80 Mbit/s This is of course evidence of the suppressed demand imposed by the single T3 link
2. Growth of the Internet
21
operating previously. The fact that usage has risen so quickly on this occasion is also indicative of the improved domestic infrastructures.. .that now exist. This quote certainly appears to support the claim that demand for bandwidth is inexhaustible. One could easily conclude that traffic essentially doubled as soon as capacity doubled. The quote is imprecise, though, since it does not say how often those “afternoon usage levels” are “in excess of 80 Mbit/s,” nor does it say how those usage levels are measured. The usage statistics for JANET, available at http://bill.ja.net/, enable us to obtain precise information. Table 2.3 shows the transfer volumes on the more heavily utilized United States to United Kingdom part of the link for several days before and after the doubling of capacity of the link. (No data for May 27 is available, and the figures for May 28, the day the second T3 was put into operation, are suspiciously low, probably reflecting incomplete measurements, so those are not included.) Table 2.3 Traffic from the United States to the JANET Network during Late Spring 1998, When the Capacity Was Doubled Day
Wed 5/20 Thu 5/21
Fri 5/22 Sat 5/23 Sun 5/24 Mon 5/25 Tue 5/26 Wed 5/27 Thu 5/28 Fri 5/29 Sat 5130
Sun 5/31 Mon 6/01 Tue 6102 Wed 6/03 Thu 6/04
Fri 6/05 Sat 6/06
Sun 6/07 Mon 6/08 Tue 6/09
GB
UtiZizution (%)
272.7 275.5 265.1 202.7 189.8 211.2 267.2
58.8 59.4 57.1 43.7 40.9 45.5 57.6
286.6 209.7 199.9 318.1 319.2 295.9 343.2 322.4 208.3 202.7 338.0 307.2
30.9 22.6 21.5 34.3 34.4 31.9 37.0 34.7 22.4 21.8 36.4 33.1
22
Kerry G. Coffman and Andrew M. Odlyzko
What we observe is that although there was substantial growth in traffic after the capacity increase, suggesting that the transatlantic link had been a bottleneck, this increase was far more moderate than the popular Internetgrowth mythology or the JANET press release would make one think. While capacity doubled, traffic increased by less than a third.
3. Growth Rates of Other Communication Services Telecommunicationshas been a growth industry for centuries, but growth rates have generally been modest, except for a few episodes, such as the beginnings of the electric telegraph (see [Odlyzko2]). For example, the number of pieces of mail delivered in the United States grew by a factor of over 50,000 between 1800 and 2000, but that was a growth rate of about 5.6% per year. (If we adjust for population increase, we find a growth rate of about 3.5% in the mail volume per capita.) The number of phone calls in the United Statesgrew by a factor of over 230 between 1900 and 2000, for a compound annual growth rate of 5.6%. (The per capita growth rate was 4.2% during this period.) Long-distance calls grew faster, about 12% per year between 1930 and 2000, and transatlantic calls faster yet. (There was just one voice circuit between the United States and Europe in 1927, when service was inaugurated. It used radio to span the ocean. This single low quality link grew to 23,000 voice circuits to Western Europe by 1995, for a compound annual growth rate of capacity of 16%.) One communications industry that has been growing very rapidly recently is wireless communication. Table 2.4 shows the growth of the U.S. cell phone industry, with the number of subscribers as of June of each year, and the revenue figures obtained by doubling those of the fmt 6 months of each year (and thus seriously understating the full-year figure). In many other countries, wireless communication has developed faster and plays a bigger role than it does in the United States. Still, even in the United States, at the end of 2000, there were close to 100 million cell phones in use, and the rate of growth was far higher than for traditional wired voice services. The cell phone example is worth keeping in mind, because it shows that volume of traffic or even the number of users has only a slight correlation to value. In the United States (unlike several other countries), there were more Internet users than cell phone subscribers at the end of 2000 (around 150 million vs. about 100 million). However, the revenues of the cell phone industry were far higher than those of the Internet. If we take a rough estimate of 60 million residential Internet users and assume they pay an average of $20 per month (both slight overestimates), we find that the total revenues from this segment come to about $15 billion. Business customers, with dedicated connections to the Internet, pay considerably less than that. For example, the 2000 revenues from business Internet connections of WorldCom (whose UUNet unit has the largest backbone in the world, often thought to carry over
2. Growth of the Internet
23
Table 2.4 Growth of the U.S.Cell Phone Industry Year
Number of Subscribers (miZlions)
Revenues (millions)
1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000
0.20 0.50 0.89 1.61 2.69 4.37 6.38 8.89 13.07 19.28 28.15 38.20 48.71 60.83 76.28 97.04
$352 721 959 1,772 2,813 4,253 5,307 7,267 9,639 13,038 17,499 22,388 26,270 30,573 38,737 49,291
30% of the total backbone traffic) were just $2.5 billion (up from $1.6 billion in 1999). The conclusion of the previous paragraph is that even in the United States, basic Internet transport revenues are less than half those of cell phones. Yet volumes of traffic are far higher on the Internet. The average daily time spent by a subscriber on a cell phone in the United States is about 8 minutes. If we count wireless communicationas taking 8 Kb/s (since compression is used), we find that the total volume of traffic generated by cell phone users in the United States at the end of 2000 was only about 1500TB/month, a tiny fraction of the 20,000 to 35,000 TB/month traffic on United States Internet backbones. (Moreover, this comparison overestimates wireless traffic, since most of the mobile calls are local, whereas backbone traffic is by definition long distance.) The comparison of revenues from Internet connectivity to those of the cell phone industry leads naturally to the next topic, namely a comparison with the entire phone industry. As we saw earlier, Internet revenues were under $25 billion in the United States in 2000. On the other hand, the revenues of the entire telephone industry (including wireless communicationand data services such as private lines leased by corporations) were around $300 billion that year. Thus, in terms of revenues, the Internet is still small. Furthermore, it is so
24
Kerry G. Coffman and Andrew M. Odlyzko
intimately tied to the phone industry that it is difficult to see what its roleis. The basic technologies (fiber transmission, SONET, and so on) that are used for Internet transport were developed initially for voice telephony, but were easily adopted for data. (Some, such as SONET, will likely turn out to be redundant, but are still widely used.) At the transport level, voice has been carried as bits for a long time. What happened is that during the late 1990s, the long-distance telecommunications infrastructure changed. It used to be dominated by the demands of voice transport, and data was a small part of what it carried. Now, however, its developmentis driven by data, especially Internet data. For quite a long time, the volume of data was extremely small, so that even though the growth rate was higher than for voice, this did not affect the overallgrowth rate of the infrastructure. That was one reason the telecommunications industry was repeatedly surprised by the demand for bandwidthin the 1990s. Moreover, the transition from voice to data domination was complicated by the presence of several types of data, with substantially different growth rates. We discuss this in more detail below. Another reason that the recent upsurge in demand for bandwidth was a surprise is that there had been several previous false predictions that data tralTic was about to explode. The excitement of the early 1990sabout the “telecommunications superhighway”and “500 channels to the home,” to be accomplished through technologies such as hybrid fiber-coax,certainly led to large financial losses and serious disappointments (see Woll21). However, there were even earlier periods of extremely rapid growth followed by sudden deceleration. For example, the number of modems in the United States grew between 1965 and 1970 at about 60% per year, to over 150,000 at the end of that period [WalkerM]. Had that growth rate been maintained, we would have had about 200 billion modems in the United States by the end of 2000, clearly an absurd number. Instead, it appears that growth in the 1970s followed the projections made around 1970 (p. 297 of [WalkerM]),which predicted annual increases of 25 to 30%. It is interestingto read the speculations in [DunnL]about the supposedly rosy prospects for electronic cash, distance education, and other data services (as well as for Picturephone) that were supposed to power the growth of networks. In general, predicting what communications services society will accept and how it will use them has been difficult (see [Luckyl, Odlyzko2]). In particular, even recent history is littered with technologies that seemed extremely promising at one point, such as ISDN (see meinrock3, WuL]) or SMDS (Switched Multimegabit Data Services-a high speed packet switched WAN technology), but never attained more than a marginal role. There are two aspects of the inability to forecast the prospects of communications technologies that are worth discussing at greater length. One goes back to the earlier discussion of wireless telephony and how the mobility offered by cell phones appears to be more important for many people than broadband Internet access. Sometimes, though, higher bandwidth did prevail. In the early days of telephony, there was widespread lack of appreciation of how attractive
2. Growth of the Internet
25
it would eventually prove to be. The telephone was used primarily for business purposes, and the telegraph appeared to be adequate for that to many. Yet it was the phone that won, even though it appeared to use bandwith very wastefully when compared to the telegraph, and even though it encouraged what was often dismissed as “idle chatter.” The attractions of instantaneous personal interactions turned out to be crucial in leading to an almost universal penetration of the telephone in industrialized countries. In the last four decades of the twentieth century though, the telecommunications industry attempted several times to extend its success with the voice telephone by introducing videotelephony. This service appeared to offer the attraction of an even deeper level of communication than voice. Yet prospective users have not only not embraced it, but have in many cases treated it with hostility. There is a growth of videoconferencing, but even that is far slower than its proponents had forecasted. For a variety of reasons that have not been completely explained, videotelephony does not appeal to people for person-to-person communication. On the other hand, mobile narrowband voice flourishes. The other aspect of the dismal record in forecasting the prospects of communications technologies that we now consider is that of the nature of traffic carried. Data networks, which in commercial settings go back about four decades, have spent essentially all this time in the shadow of the much larger voice telephone network. (They also benefited from being able to use the infrastructure of the phone network, and were also constrained by its limitations, but that is less relevant for us here.) It was therefore natural for networking experts to continuously think of voice traffic, and in particular of the possibility of eventually carrying it as data. Looking further out, to a stage where the progress of technology appeared to offer the possibility of data networks becoming much larger than the phone networks, it was also natural to think of enriching the communicationsmedium through the addition of video. (See the projections of Estill Green [Green, Lucky21 and Hough [Hough], for example.) Later, the huge volume of broadcast data (radio and especially television) offered further possibilities for traffic that could be carried on data networks. The key point is what was seen as eventually filling data network was streaming multimedia traffic. The Internet’s rise to dominance was a surprise for many reasons, but one of the main ones was that it did not fit this model. Although much current work on Internet technologies is devoted to streaming multimedia, there are good reasons, to be discussed later, why such traffic is not likely to dominate. Althoughit has proven difficult to forecast which technologieswill be widely adopted, once a service had been successfullyintroduced, it often showed regular growth rates for extended periods of time [Odlyzko2].The approximately 30% annual growth rate that had been projected in 1970 for data transmission (or, to be more precise, for the proxy for actual transmission that is offered by the number of modems) appears to have held not just in the 1970s, but in the 1980s and most of the 1990s as well. There are no comprehensive
26
Kerry G. Coffman and Andrew M. Odlyzko
statistics (and there are measurement problems, in that private lines, whose bandwidth is often taken as a measure of the data traffic, can also be used for voice transmission). However, there are a few pieces of evidence supporting those growth rates around 1980 in [deSolaPITH]. Those same growth rates appeared to also hold for long-distance private line transmission in the mid 1990s [CoffmanOl] and for local data bandwidth in the late 1980s and most of the 1990s [Galbi]. The comprehensive data summarized in [Galbi] is especially interesting. During the late 1980s and most of the 199Os, installed computer power came close to doubling each year, and the new “Information Economy” was taking root, but this was not reflected in the volume of data traffic. This low rate of growth in data transmission may have come from the high cost and poor quality of data transmission or from other causes, such as lack of uniform standards that would enable easy data communication between companies. It may also have been caused to a large extent by the slow rate at which computation and communication technologies were adopted. Whatever the reasons, this low growth rate of approximately 30% a year (low by comparison to growth of computing power) in data transmission was higher than that of voice networks. Hence by the mid 199Os, the bandwidth of long-distance data networks (primarily private lines used for intracompany communication) was already comparable to that of the voice network [CoffmanOl]. The Internet has historically had a growth rate of close to 100% per year in the traffic it carried. As Table 2.1 shows, it was growing with striking regularity in the early 1990s at this rate. Then it experienced a period of astronomical growth in 1995 and 1996, and then reverted to an approximate doubling each year in 1997, and has continued growing at about that rate through the end of 2000. The big question is how fast it will grow in the future. While the overwhelming preponderance of opinion all through the end of 2000 was that Internet traffic was doubling every 3 or 4 months, by early 2001 the consensus started changing. Some analysts even began projecting declines in the growth rates to the 50% per year range by around 2005. And indeed, some sources of growth did dry up. With the crash of telecom stocks (caused largely by the realization that expected demand and revenues were not materializing), investments slowed, and many dot-coms that had been busily filling transmission pipes with their content disappeared. In a related development, corporate managements started asking for detailed justifications for new data networking expenditures instead of rushing to endorse any proposals that came along. At various enterprises, the growth rates of data traffic, which had been close to doubling every year in the late 199Os, began to slow down toward doubling every 18 or 24 months. It is not inconceivable that overall data traffic growth may be moving back to its historical rate of around 30% per year. We do not think this will occur, but before considering the reasons why (presented in detail in Sections 6 to 9), we look at the general history of the Internet and its growth rates.
2. Growth of the Internet
27
At this point we just remark that the dominant role of the Internet in communications, whether in terms of bandwidth of networks or popular consciousness, is a fairly recent phenomenon. There had been extensive discussions of the “Information Superhighway” and the “National Information Infrastructure” for a long time. Leading thinkers foresaw the possibilities for much improved communication offered by new technologies, and there was tremendous effort devoted to various systems. However, the general expectation was that the “Information Superhighway” would be composed of a very heterogeneous collection of (interconnected)networks. This was true even as late as the beginning of the Clinton presidency in 1993 and 1994 (see [NII]). It was only in the mid to late 1990s that the Internet was perceived as evolving toward an all-encompassing network, carrying all types of traffic.
4. Internet History Over the past 5 to 10 years, we have witnessed not only an explosion of activity, but the creation of entirely new sectors within the optical industry. As the concept of wavelength division multiplexing (WDM) began to emerge, many new companies developing WDM transport equipment came into existence. The newer enterprises pushed the older established equipment vendors to more aggressive deployment schedules, and a constant downward trend for the corresponding prices of WDM transport equipment followed. In what appeared to be an almost insatiable demand for more bandwidth, a situation arose that allowed the creation of the new companies and the accompanying innovation. Not only did new equipment vendors emerge, but also new national-scale carriers were created. This trend is continuing as the concept of optical layeringhetworking is gaining acceptance and new optical equipment companies are being formed on a regular basis. They deal not only with “traditional” WDM transport equipment, but also with terrestrial ultra long-haul systems, regional and metro optimized systems, and various incarnations of optical cross connects. There were hundreds of developments and contributions enabling this burst of activity. Many of the technical innovations are described in this book and its predecessors. However, perhaps the greatest single factor that fueled this phenomena was the belief and perception that traffic, and hence needed capacity, were growing at explosive rates. This is a remarkable fact, especially when one recalls that around 1990 both the traditional carriers and most of their equipment vendors still expected the traffic demands to not vary much from the voice demand growths (which historically was around 10% per year). In fact, both carriers and equipment vendors were arguing that WDM would not be needed and that going to individual channel rates of at most 10 Gb/s would be adequate. Also, around 1995, the conventional wisdom was that 8-channel WDM systems would suffice well into the foreseeable future. Now it almost
28
Kerry G. Coffman and Andrew M. Odlyzko
appears as if the pendulum has swung the other way. Is too much capacity being deployed?Are many of the reported traffic growth rates correct? And if so, will they continue? As we explainedin the previous section, the early skepticism about the need for high-capacity optical transport was rooted in the reality of the telecommunications networks. Up until 1990, they were dominated by voice, which was growing slowly. Then, by the mid 1990s, they came to be dominated (in terms of capacity) by private lines, which were growing three or four times as fast. And then, in the late 1990s, they came to be dominated by the Internet, which was growing faster still. Before we go through the analyses for the tr&c growth on the Internet, we must first at least define the Internet and describe its history and structure. This is paramount in helping put much of the later described growth analyses into perspective. When one now speaks of the Internet, it is usually described as an evolution from ARPANET to NSFNet, and finally to the commercial Internet that now exists. Arguably, the phenomenal growth of the Internet started in 1986 (more than 17 years after its “birth”) with NSFNet. However, the path was very complicated and full of many twists and turns in its roughly 40-year history [Cerf, Hobbes, Leiner]. From the very early research in packet switching, academia, industry, and the U.S. government have been intertwined as partners. Ironically, the beginnings of the Internet can trace itself back to the Cold War and specihlly to the launch of Sputnik in 1957. The U.S. government formed the Advanced Research Project Agency (ARPA; the name was later changed to DARPA, Defense Advanced Research Project Agency, and later back to ARPA) the year after the launch with the stated goal of establishing a U.S. lead in technology and science (with emphasis on military applications). As ARPA was establishing itself, there were several pivotal works [Kleinrockl, Baran] in the early 1960s on packet switching and computer communications. These works and the efforts they spawned laid many of the foundations that enabled the deployment of distributed packet networks. J. C. R. Licklider (of MIT) [LickC] wrote a series of papers in 1962 in which he “envisioned a globally interconnected array of computers which would enable ‘everything’ to easily access data and programs from any of the sites.” Generically speaking, this idea is not much diflterent from what today’s Internet has become. Of importance is the fact the Licklider was the first head of the computer research program at DARPA (beginning in 1962), and in this role he was instrumental in pushing his concept of networks. Kleinrock published both the first paper on packet switching and the first book on the subject. In addition, Kleinrock convinced several key players of the theoretical feasibility of using packets instead of circuits for communications. One such person was Larry Roberts, one of the initial architects for the ARPANET. In the 1965 to 1966time frame, ARPA sponsoredstudies on a “cooperativenetwork of [users]
2. Growth of the Internet
29
sharing computers” [Leiner], and the first ARPANET plans were begun, with the first design papers on ARPANET being published in 1967. Concurrently, the National Physical Laboratory (NPL) in England deployed an experimental network called the NPL Network that made use of packet switching. It utilized 768 Kb/s lines. A year before the moon landing, in 1968, the first ARPANET requests for proposals were sent out, and the first ARPANET contracts were awarded.Two of the earliest contractswent to UCLA to develop the Network Measurement Center, and to Bolt, Beranek, and Newman (BBN) for the Packet Switch contract (to construct the Interface Message Processors or IMPs-effectively the routers). Kleinrock headed the Network Measurement Center at UCLA and it was selected as the first node on the ARPANET. The first IMP was installed at UCLA and the first host computer was connected in September of 1969. The second node was at Stanford Research Institution (SRI). Two other nodes were added at UC Santa Barbara and in Utah, so that by the second half of 1969,just months past the lkst moon landing, the initial four-node ARPANET became functional. This was truly the initial ARPANET, and thus a case can be made that this was when the Internet was born. The first message carried over the network went from Kleinrock’s lab to SRI. Supposedly the first packet sent over ARPANET was sent by Charley Kline, and as he was trying to log in the system crashed as the letter “ G of “LOGIN” was entered. One of the next major innovations for the fledgling Internet (i.e., ARPANET) was the introduction of the first host-to-host protocol, called Network Control Protocol, or NCP, which was first used in ARPANET in 1970. By 1972, all of the ARPANET sites had finished implementing NCP. Hence the users of ARPANET could finally begin to focus on the development of applications-another paramount driver for the phenomenal growth and sustained growth of the Internet. It was also in 1970 that the first crosscountry link was established for ARPANET by AT&T between UCLA and BBN (at the blinding rate of 56 Kbh). By 1971, the ARPANET had grown to 15 nodes and had 23 hosts. However, perhaps the most influential work that year was the creation of an e-mail program that could send messages across a distributed network. (E-mailwas not among the original design criteria for the ARPANET, and its success caught the creators of this network by surprise.) Ray Tomlinson of BBN developed this application, and his original program was based on two previous ones [Hobbes]. Tomlinson modified his program for ARPANET in 1972, and at that point its popularity quickly soared. In fact, it was at this time that the symbol “@,, was chosen. Arguably, Internet e-mail as we know it today can trace its origins directly to this work. Internet e-mail was clearly one of the key drivers for the popularity (and hence the phenomenal traffic-growthdemands) of the Internet and was the first “killer app” for the Net. It was every bit as critical to the Internet’s “success” as spreadsheet applications were to the popularization of the PC. Internet e-mail provided
30
Kerry G. Coffman and Andrew M. Odlyzko
a new model of how people could communicate with each other and alter the very nature of collaborations. Although there was already considerable work being done on packet networks outside the United States, the first international connections to the ARPANET (to England via Norway) took place in 1973. To put the time frame in perspective, this was the same year that Robert Metcalfe did his PhD that described his idea for Ethernet. Also during this year, the number of ARPANET c‘users”was estimated to be 2,000 and that 75% of all the ARPANET t r a c (in terms of bytes) was e-mail. One needs to note that in only 1 to 2 years from its introduction onto the Internet, e-mail became the predominant type of traffic. The same behavior took place several years later for HTML (Hypertext Markup Language) (i.e., Web traffic), and to a somewhat lesser degree, this was seen for Napster-like traffic within many networks a few years later. Several other key developments began to take place in the mid 1970s. The initial design specification for TCP (Transfer Control Protocol) was published by Vint Cerf and Bob Kahn in 1974 [CerfK]. The NCP protocol, which was being utilized at the time, tended to act like a device driver, whereas the future TCP (later TCP/IP) would be much more like a communications protocol. As is discussed later, the evolution from ARPANET’s NCP protocol to TCP (which in 1978 was split into TCP and IP (Internet Protocol)) was critical in allowing the future growth and scalability of today’s Internet. DARPA had three contracts to implement TCPLIP (at the time still called TCP), at Stanford (led by Cerf), BBN (led by Ray Tomlinson), and UCLA (led by Kirsten). Stanford produced the detailed specification and within a year there were three independent implementations of TCP that could interoperate. It is noted that the basic reasons that led to the separation of TCP (which guaranteed reliable delivery) from IP actually came out of work that was done trying to encode and transport voice through a packet switch. It was found that a tremendous amount of bdering was needed in order to allow for the appropriate reassembly after transmission was completed. This in turn led to trying to find a way to deliver the packets without requiring a guaranteed level of reliability. In essence, the UDP (User Datagram Protocol) was created to allow users to make use of IP. In addition, it was also in 1978 that the first commercial version of ARPANET came into existence when BBN opened Telenet. In 1981-1982, the first plans were made to “migrate”from NCP to TCP. It is claimed by some that it was this event (TCP was establishedas the protocol suite for ARPANET) was truly the birth of the InternetAefined as a connected set of networks, specifically those with TCP/IP. A few years later (in 1983) another major development occurred, which later enabled the Internet to scale with the “explosive” growth and popularity of the future Internet. This was the development of the name server, which evolved into the DNS [Cerf, Leiner]. The name server was developed at the University of Wisconsin [Hobbes]. This made it easy for people to use the network because hosts were assigned names
2. Growth of the Internet
31
and it was not necessaryto remember numeric addresses. Much of the credit for the invention of the DNS (Domain Name Server) is given to Paul Mockapetris of USC/ISI [Cerfl. The year 1983 was also the date for two other key developments on ARPANET. The first one was the cutover from NCP to TCP on the ARPANET. Secondly, ARPANET was split into ARPANET and MILNET. Although the road was convoluted, this split was one of the key bifurcation points that later allowed NSFNet to come into existence. Soon thereafter (in 1984), the number of hosts on ARPANET had grown to 1,000, and the next year in 1985 the first registered domain was assigned in March. In 1985, NSFNet was created with a backbone speed of 56 Kb/s. Initially, there were five supercomputing centers that were interconnected. One of the paramount benefits of this was that it allowed an explosion of connections (most importantly from universities) to take place. Two years later in 1987, NSF agreed to work with MERIT Network to manage the NSFNet backbone. The next year (1988), the process of upgrading the NSFNet backbone to one based on T1 (Le., 1.5 Mb/s links) was begun. In 1987, the number of hosts on the Internet broke 10,000. Two years later in 1989, this had grown to around 100,000, and 3 years after that, in 1992, it reached the 1,000,000 value. It is noted that if you look at how the number of hosts had been growing from 1984 to 1992, that it was still pretty much tracking a growth curve that was less than tripling each year (i.e., doubling every 9 months). In the 1985-1986 time frame, a key decision was made that had very long-term impact: that TCP/IP would be mandatory for the NSFNet program. In the 1988-1990 time frame, a conscious decision was made to connect the Internet to electronicmail carriers, and by 1992,most of the commerciale-mail carriers in the United States were “like the Internet.” This was still another development that cemented e-mail as the single most important application to take advantage of the Internet. In 1990, the ARPANET ceased to exist, and arguably NSFNet was the essence of the Internet. The following year, commercial Internet Service Providers (ISPs) began to emerge (PSI, ANS, Sprint Link, to name a few), and the Commercial Internet Xchange (CIX) was organized in 1991 by commercial ISPs to provide transfer points for traffic. NSF’s lifting the restriction on the commercial use of the Net was again one of the pivotal decisions. This was again a key bifurcation point, in that this helped set the stage for the complete commercialization of the Net that would follow only a few years later. In 1991, the upgrading of the NSFNet backbone continued as the work to upgrade to a T3 (Le., 45 Mb/s links) began. It is also interesting to note that it was the next year, 1992, that the term “surfing the Internet” was first coined by Jean Armour Polly [Polly], only 2 years before the ARPANEThternet celebrated its 25th anniversary. It was in the 1993-1995 time period that several major events seemed to emerge that fueled an almost explosivegrowth in the popularity of the Internet.
32
Kerry G. Coffman and Andrew M. Odlyzko
One of the key ones was the introduction of “browsers,” most notably Mosaic. This led to the creation of Netscape, which went public in 1995. Even as early as 1994, WWW (i.e., predominantly HTML) tr&c was increasing in volume on the Net. By then it was the second most popular type of traffic, surpassed only by FTP (File Transfer Protocol) tr&c. However, in 1995 WWW tra& surpassed FTP as the greatest amount of tr&c. In addition, the traditional online dial-up systems such as AOL (America Online), Prodigy, and CompuServe began to provide Internet access. In 1996, the Net truly became public with the NSFNet being phased out. Soon thereafter, major infrastructure improvements were made within the transport part of the Internet. The Internet began to upgrade much of its backbone to OC3-0C12 (up to 622Mb/s) links, and in 1999, upgrades began for much of the Net to OC48 (2.5 Gb/s) links.
5. The Many Internet Growth Rates The Internet is very hard to describe. By comparison, even the voice phone system, which is a huge enterprise, far larger in terms of revenues than the Internet, is much simpler. In the phone system, the basic service is well defined and simple to describe. The users have only limited ability to interact with the system. The Internet is completely different. Users interact with the system in a multiplicity of ways, on widely different time scales, and there are many complicated feedback loops. The paper [FloydP] is an excellent overview of the problems that arise in attempting to simulate the Internet. The problems of measuring the Internet are also formidable. There are many different measures that are relevant. In this chapter,just as in the papers [CoffmanOl,CoffmanO2], we will concentrate on traffic as measured in bytes. For the optical fiber telecommunications industry, it is capacity that is most relevant. Unfortunately, there are numerous problems in measuring capacity. Much of the fiber is not lit, and even when it is lit, often only a few wavelengths are lit. Finally, much of the potential capacity is used for restoration, through SONET or other methods. In addition, even at the levels of links used for providing IP traffic, it is hard to obtain accurate capacity measurements, because few carriers provide detailed data. Further, this type of capacity has a tendency to jump suddenly, as bandwidth is usually increased in large steps (such as going from OC3 to OC12, and then 0048, a phenomenon that contributes to the low utilization of data links [Odlyzkol]). Thus, there is little regularity in capacity growth figures. On the other hand, we do find astonishing regularity in traffic growth, which leads us to propose that a form of Moore’s Law applies. In the long run, we expect that capacity will grow slightly faster than traffic, as we explain later. For many purposes other measures are important, such as the number of users, how they spend their time, how many and what types of commercial
2. Growth of the Internet
33
transactions they engage in, and so on. There are many sources of such data, and useful references can be found at [Cyberspace, MeekerMJ, Nua].
6. Internet Traffic and Bandwidth Growth Whether Internet traffic doubles every 3 months or just once a year has huge consequences for network design as well as the telecommunicationsequipment industry. Much of the excitement about and funding for novel technologies appears to be based on expectations of unrealistically high growth rates ([Bruno]). In this section we briefly examine avariety of examples in an attempt to understand the traffic-growth rates that the Internet has experiencedover its lifetime. There are places where the traffic is growing at rates that exceed 100% per year. One such example is LINX (London Internet Exchange). Its online data, available at http://ochre.linx.net/, clearly shows a growth rate of about 300% from early 1999 to early 2001. There are also examples of even higher growth rates, although those tend to be for much smaller links or exchange points. However, there are also numerous examples of much more slowly growing links. In this section we briefly present growth rates from avariety of sources and attempt to put them into context. In an earlier study [CoffmanOl] in 1997, we found that the evidence supported a traffic growth rate of about 100% per year (doubling annually). Four years later, the general conclusion is that Internet traffic still appears to be growing at about 100% per year. In other words, we have not found any substantial slowdown in the growth rate. Some recent reports and projections conclude that Internet traffic is only about doubling each year, but claim that it was growing much faster until recently, and that its growth rate will continue to slow down. In that view, the telecom crash of 2000 was associated with a sudden decline in the growth rate of traffic. As far as we can tell, that is not accurate. The general rate of growth of traffic appears to have been remarkably stable throughout the period 19972000. As one of the most convincing pieces confirming this claim, we cite the news story ([Cochrane]) based on official figures from Telstra, the dominant Australian telecommunications carrier. This story reports that Telstra’s IP traffic was almost exactly doubling each year between November 1997 and November 2000. (The printed version of this news story, but not the one available online at the URL listed in [Cochrane], shows a very regular growth, about 100% per year, from the beginning of 1997 to November 2000.) Hence our conclusion is that the problems the photonics industry is experiencingare not caused by any sudden slowdown in traffic, but rather by a realization that the astronomical growth rates that people had been assuming were fantasies. Most of this section is drawn from the more detailed account in [Coffman02]. There are only a few new pieces of information. For example, the China Internet Network Information Center has statistics (at www.cnnic.net.cn/develst./e-index.shtm1)of the Internet bandwidth between
34
Kerry G. Coffman and Andrew M. Odlyzko
China and the rest of the world. It grew from 84.64Mb/s in June 1998 to 2,799 Mbls in December 2000, for a compound growth rate of 305% per year. Thus, even in a rapidly growing economy like that of China, where Internet penetration is low and is trying to catch up with the industrialized world, traffic is only doubling about every 6 months. The comparison of the international bandwidth for Australia and China is instructive. In December 2000, Telstra had about 1,000Mb/s to the rest of the world, about a third of Chinese bandwidth. Thus, making allowances for other Australian carriers, we can speculate that Australia may be exchanging half as much traffic with international destinations as China does, even though the latter has over 60 times the population. This shows the degree to which countries can differ in their intensity of Internet usage. The data in [Cochrane], showing that Telstra’s IP traffic in November 2000 reached about 270 TBlmonth, also shows that our general estimates for U.S. backbone traffic are reasonable, because the United Statesis not only larger than Australia, but also richer on aper capita basis and has a better developed telecommunications infrastructure. In the remainder of this section we examine some of the data and trends from ISPs, exchange points, and residential traffic patterns, along with traffic from “stable sources,” such as corporate, research, and academic networks. It is noted that the data for the first two sources (ISPs and exchange points) are not nearly as complete nor reliable as only a few years ago. However, much better data are available for the “stable sources,” and several are examined in much more detail later. As a brief note on conversion factors, traffic that averages 100Mb/s is equivalent to about 30 TB/month. (It is 32.4 TB for a 30-day month, but such precision is excessive given the uncertainties in the data we have.) Unfortunately, the largest ISPs do not release reliable statistics. This situation was better even a couple of years ago. Much of the older data was used in previous studies ([CoffmanOl]).For example, MCI used to publish precise data about the traffic volumes on their Internet backbone. Even though they were among the first ISPs to stop providing official network maps, one could obtain good estimates of the MCI Internet backbone capacity from public presentations. These sources dried up when MCI was acquired by WorldCom, and the backbone was sold to Cable &Wireless. As was noted in [CoffmanOl], the traffic-growthrate for that backbone had been in the range of 100% a year before the change. Today, one can obtain some idea of the sizes (but not trafEc) of various ISP networks through the backbone maps available from Boardwatch. However, even those are not too reliable. The only large ISP in the United States to provide detailed network statistics is AboveNet, at http:lluww.above.netltrafficl. Therefore, we looked at this ISP in moderate detail. We have recorded the MRTG (Multi-RouterTraffic Grapher)[MRTG]data for AboveNet for March 1999, June 1999, February 2000, June 2000, November 2000, and April 2001.
2. Growth of the Internet
35
The average utilizations of the links in the AboveNet long-haul backbone during those 4 months were 18, 16, 29, 12, 11, and lo%, respectively. (The large drop between February and June 2000 was caused by deployment of massive new capacity, including four OC48s. One of the reasons we concentrate on traffic and not network sizes in this chapter is that extensive new capacity is being deployed at an irregular schedule and is often lightly utilized. Thus, it is hard to obtain an accurate picture of the evolution of network capacity.) If one just adds up the volumes for each link separately, one finds that between March 1999 and April 2001, the total volumes of traffic increased at an annual growth rate of about 200%. However, this figure has to be treated with caution, as actual traffic almost surely increased less than 200%. During this period, AboveNet expanded geographically, with links to Japan and Europe, so that at the end it probably carried packets over more hops than before. Because we are interested in end-to-end traffic as seen by customers (which can be thought of as the ingress and/or egress traffic into and/or out of “the network”), we have to deflate the sum of traffic volumes seen on separate backbone links by the average number of hops that a packet makes over the backbones (perhaps around three). Even when there is reliable data for a single carrier, such as AboveNet, some of the growth seen may be coming from gains in market share, both from gains within a geographical region and from greater geographical reach, and not from general growth in the market. We next look at Internet exchangepoints. When the NSF Internet backbone was phased out in early 1995, it was widely claimed that most of the Internet backbone traffic was going through the Network Access Points (NAPs) (which are effectivelyinterconnectionvehicles), which tended to provide decent statistics on their traf3ic. Currently it is thought that only a small fraction of backbone traffic goes through the NAPs, while most goes through private peering connections. Furthermore, NAP statistics are either no longer available or not as reliable. This is in sharp contrast to the situation in 1998 [CoffmanOl]. As documented elsewhere [CoffmanO2], there is very little that can be reliably concluded about current growth rates of Internet traffic by examining the statistics of the public NAPs in the United States. However, the situation was slightly better when we examined a large number of international exchange points. These included LINX, AMS-IX (the Amsterdam Internet exchange), the Slovak Internet exchange, HKIX (a commercial exchange created by the Chinese University of Hong Kong, BNIX (located in Belgium), the INEX (an Irish exchange), and FICX (the Finish exchange). Some of these show growth rates of only about doubling per year while others show much faster growth rates. Trafiic interchange statistics are hard to interpret, unless one has data for most exchanges, which is virtually impossible to obtain. Much of the growth one sees can come from ISPs moving from one exchange to another, moving their traffic from one exchange to another, or coming to an exchange in preference to buying transit from another ISP. Consider the specific case of LINX. A large part of its growth
36
Kerry G. Coffman and Andrew M. Odlyzko
is almost surely caused by more ISPs exchanging their traffic there. Between March 1999 and March 2000, the ranks of ISPs that are members of LINX have grown by about two-thirds, based on the data on the LINX home page. Hence the averageper-member traffic through LINX may have increased only around 120% during that year. The traffic from residential U.S. customers will probably begin to increase at a faster rate in the near future. The growth in the number of users is likely to diminish as we reach saturation. (You cannot doublethe ranks of subscribersif more than half the people are already signed up!) However, broadband access, in the shape of cable modems and DSL (and to a lesser extent fixed wireless links), will stimulate usage. The evidence so far is that users who switch to cable modem or DSL access increase their time online by 50 to loo%, and the total volume of data they download per month by factors of five to ten. A five- or tenfold growth in data traEic would correspond to a doubling of traffic every 4 months if everyone were to switch to such broadband access in a year. However, that is not going to happen. At the end of 1999, there were about 3 million households in the United States with broadband access. The most ambitious projections for cable modem and DSL access call for about 13 million households to have such links in 2003, and between 5 M O million in the year 2007. That is approximatelya doubling each year. (There was apparently almost a tripling in the ranks of households with broadband access in 2000, but the telecom crash that wiped out many of the ADSL providers has led to a slowdown in the pace of deployment in 2001.) The traffic from a typical residential broadband customer is likely to grow beyond the level we see today as more content becomes available and especially as more content that requires high bandwidth is produced. Still, it is hard to see average traffic per customer among those with broadband connections growing at more than 50% a year. Together with a doubling in the ranks of such customers, this might produce a tripling of traffic from this source. Because the ranks of customers with regular modems are unlikely to decrease much, if any, and because their traffic dominates, it appears that the most likely scenario will be for the total residential customer traffic to grow no faster than 200% per year, and probably closer to 100% per year. (Access from information appliances,which are forecast to proliferate, is unlikely to have a major impact on total traffic, since the mobile radio link will continue to have small bandwidth compared to wired connections.) We next consider traf€ic at various stableinstitutions-corporate, academic, and governmental. Growth in traffic can be broken down into growth in the number of traffic sources and growth in traffic per source. For LINX, much of the increase in traffic may be coming from an increase in member ISPs. For individual ISPs, much of the increase in traffic may also be coming from new customers. Yet in the end, that kind of growth is limited, as the market becomes saturated. The rest of this section focuses on rates of growth in traffic from stable sources. Now nothing is completely stable, as
2. Growth of the Internet
37
the number of devices per person is likely to continue growing, especially with the advent of information appliances and wireless data transmission. Hence we will consider growth in traffic from large institutions that are already well wired, such as corporations and universities. Most corporations do not publicize information about their network traffic, and many do not even collect it. However, there are some exceptions. For example, Lew Platt, the former CEO of Hewlett-Packard, used to regularly cite the HP intranet in his presentations.. The last such report, dated September 7, 1998, and available at http://www.hp.com/financials/textonly/personnel/ceo/~es.ht~, stated that this network carried 20 TB/month, and a comparison with previous reports shows that this volume of traffic had been doubling each year for at least the previous 2 years. (As an interesting point of comparison, the entire NSFNet Internet backbone carried 15TB/month at its peak at the end of 1994.) Several other corporations have provided data showing similar rates of growth for their Intranet trafllc, although some indicated their growth has slowed, and a few have had practically no recent growth. Internal corporate tr&c appears to be growing much more slowly than public Internet traffic. Data for retail private lines as well as for Frame Relay and ATM (Asynchronous Transfer Mode) services show aggregate growth in bandwidth (and therefore most likely also traffic) in a range of 3040% per year. The growth is slow for retail private lines and fast for Frame Relay and ATM. These rates are remarkably close to the growth rate observed in the late 1970s in the United States, which was around 30% per year [deSolaPITH]. Thus, it is the corporate traffic to the public Internet that is growing at 100% per year. It is also important to note that in the year 2000, over two-thirds of the volume on the public Internet appeared to be business to business. Thus, the accelerationof the overallgrowth rate of data trafficto about 100%per year from the old 30% or so a year appears to be a consequence of the advantages of the Internet, with its open standards and any-to-anyconnectivity. For the remainder of this section we concentrate on publicly available information, primarily about academic, research, and government networks. These might be thought of as unrepresentative of the corporate or private residential users. Our view is just the opposite, in that these are the institutions that are worth studyingthe most, since they normally alreadyhave broadband accessto the Internet, tend to be populated by technically sophisticated users, and tend to try out new technologies first. The spread of Napster through universities is a good example of the last point. We believe that Napster and related tools, such as Gnutella and Wrapster, are just the forerunners of other programs for sharing of general information, and not just for disseminating pirated MP3 files. As we explained elsewhere, there is already much more digital data on hard disks alone than shows up on today’s Internet. Further, this situation is likely to continue. The prevalent opinion appears to be that in data networks, “If you build it, they will fill it.” Our evidence supports this, but with the important
Kerry G. Coffian and Andrew M. Odlyzko
38
qualiiication that “they” will not fill it immediately. That certainly has been the experience in local area networks, LANs. The prevalence of lightly utilized long-distance corporate links was noted in [Odlyzkol]. That paper also discussed the vBNS (very High Speed Backbone Network) research network, which was extremely lightly loaded. Here we cite another example of a large network with low utilizations and moderate growth rates. Abilene is the network created by the Internet2 consortium of U.S. universities. Its backbone consists of 13 OC48 (2.4 Gb/s) links. Moreover, most of the consortium members had OC3 links to it. The average utilization in June 2000 was about 1.5%, and by April 2001 it had grown to about 4.1%. Thus, in spite of the uncongested access and backbone links, tr&c did not explode. Even on more congested links, it often happens that an increase in capacity does not lead to a dramatic increase in traffic. This is supported by several examples. Such examples include the University of Waterloo, the SWITCH network, the NORDUNet network, the European TEN-155 network, the Merit network, the University of Toronto, Princeton University, and the University of California at Santa Cruz [CoffmanO2]. Later we go into moderate detail for these networks. Figure 2.1 shows statistics for the traffic from the public Internet to the University of Waterloo over the last 7 years. Detailed statistics for the Waterloo network are available at http://www.ist.uwaterloo.ca/cn/#Stats,but Fig. 2.1 is based on additional historical data provided to us by this institution. Just as for the JANET network discussed previously and the SWITCH network to be discussed later, as well as most access links, there is much more traffic from the public Internet to Traffic from the Internet to the University of Waterloo 03
I
1993
1994
1995
I
I
I
1996
1997
1998
1999
2000
year
Fig. 2.1 T r a c on the link from the public Internet to the University of Waterloo. The line with circles shows average traffic during the month of heaviest traffic in each school term. The step function is the full capacity of the link.
2. Growth of the Internet
39
the institution than in the other direction. Hence we concentrate on this more congested link, because it offers more of a barrier. We see that even substantial jumps in link capacity did not affect the growth rate much. Traffic has been about doubling each year for the entire 7-year period. (Overall, the growth rate at the University of Waterloo has slowed, about 55% from early 1999 to early 2000. This was at least partially the result of official limits on individual users that were imposed, limits we will discuss later.) The same phenomenon of traffic doubling each year, no matter what happens to capacity, can be observed in the statistics for the SWITCH network, which provides connectivity for Swiss academic and research institutions. The history and operations of this network are described in [Harms, ReichlLS], and extensive current and historical data are available at http:// www.switch.ch/lan/stat/. The data used to prepare Fig. 2.2 was provided to us by SWITCH. As is noted in [ReichlLS], the transatlantic link has historically been the most expensive part of the SWITCH infrastructure, and at times was more expensive than the entire network within Switzerland. It is therefore not surprising that this link tends to be the most congested in the SWITCH network. Even so, increasing its capacity did not lead to a dramatic change in the growth rate of traffic. If we compare increases in volume of data received between November of one year and January of the following year, there was an unusually high jump (420/0)from November 1998to January 1999.This was in response to extreme congestion experienced at the end of 1998, congestion that produced extremely poor service, with packet loss rates during peak periods exceeding 20%. However, over longer periods of time, the growth rate has been rather steady at close to 100% per year and independent of the capacity
H I
1996
Capacity and traffic on SWITCH transatlantic link
1997
1998
1999
2000
2001
32
year
Fig. 2.2 Capacity of link between the Swiss SWITCH network and the United States and traffic on it toward Switzerland.
40
Kerry G. Coffman and Andrew M. Odlyzko
of the link. More detailed data about other types of SWITCH traffic can be found at http://www.switch.cMan/stat/ through the “Public access” link. The listings available there as of mid 2000, as well as those from previous years, show that various transmissions tended to grow at 100 to 150% per year. It is worth noting that capacity grew faster than traffic, but not too much faster. Merit Network is a nonprofit ISP that serves primarily Michigan educational institutions.It has data availableonline at http://www.merit.net/michnet/ statistics/direct.htmlthat goes back to January 1993. This data was used to construct the graph in Fig. 2.3. The data for January 1993 through June 1998 shows only the number of inbound IP packets. The data for months since July 1998 is more complete, but it is so complete, with details of so many interfaces, that we have not yet determined the best way to use it. Hence we have used only the earlier information for January 1993 through June 1998. The resulting time series is a reasonable, although imperfect, representation of a straight line, modulated by the periodic variations introduced by the academic calendar. The growth rate is almost exactly 100% per year. The research networks that were examined have low utilizations. It should be emphasized that this is not a sign of inefficiency. Many novel applications required high bandwidth to be effective. That, along with some additional factors, such as the high growth rate, lumpy capacity, and pricing structure, contributes to the much lower utilization of data networks than of the longdistance voice network [Odlyzkol]. The general conclusion that can be drawn from the examples listed in this section (along with numerous other examples) is that data traffic has a remarkable tendency to double each year. There are of course slower and faster growth rates. Overall though, they tend to cluster in the Vicinity of 100% per year. MichNet traffic
1993
1994
1995
1996
1997
1998
year
Fig. 2.3
Traffic from Merit Network to customers
1999
2. Growth of the Internet
41
To date, the authors have not seen any large institutions with traffic doubling anywhere close to every 3 or even 4 months. The growth rates that are cited here are often affected strongly by restrictions imposed at various levels. As described elsewhere [CoffmanOl, CoffmanO2], some of the explicit limits are imposed by network administrators. The arrival of Napster (discussed in Section 7) led many institutions to either ban its use or else limit traffic rates to some parts of the campus (typically student dormitories). Push technologies were stifled at least partially because enterprise network administrators blocked them at their firewalls. E-mail often has size restrictions that block large attachments (and in some cases all attachments are still banned). Teleconferencing is only slowly being experimented with on corporate intranets, and even packetized voice sees very limited (although growing) use. Similar constraints apply to most of the content seen on the Web. As long as a large fraction of potential users have limited bandwidth, such as through dial modems, managers of Web servers will have an incentive to keep individual pages moderate in size. Thus, one can see that Internet traffic is subject to a variety of constraints at different levels. Some are applied by network managers, others by individual users, and the interaction of these constraints with the rising demands is fundamental in understanding what produces the growth rates observed. The ability to sustain the high growth rate of Internet traffic will require the creation of new applications that will generate huge volumes of traffic. At current growth rates, by 2005 there will be eight times as much Internet as voice traffic (on the U.S. long-haul networks). If voice were packetized, in all likelihood the voice traffic would only account for about 3% of the Internet traffic. Thus, voice traffic will not fiU the pipes that are likely to exist, and neither will traditional Web surfing. This will create a dilemma for service providers, network administrators, and equipment suppliers: To sustain the growth rates that the industry has come to depend on, and to accommodate the progress in technology, new technologies are needed. Such applications will appear disruptive to network operations today, and as such, they often have to be controlled. However, in the long run, they must be encouraged.
7. Disruptive Innovation It is often said that everything changes so rapidly on the Internet that it is impossible to forecast far into the future. The next “killer app” could disrupt any plans that one makes. Yet there have been just two “killer apps” in the history of the Internet: e-mail and the Web (or, more precisely, Web browsers, which made the Web usable by the masses). Many other technologies that had been widely touted as the next “killer app,” such as push technology have fizzled. (Push technology allows the sending of information directly to one’s
42
Kerry G. Coffman and Andrew M. Odlyzko
computer instead of the computer needing to actively go out and obtain it.) Furthermore, only the Web can be said to have been truly disruptive. From the first release of the Mosaic visual browser around the middle of 1993, it apparently took under 18 months before Web trafKc became dominant on Internet backbones. It appears overwhelmingly likely that it was the appearance of browsers that then led, in combination with other developments, to that abnormal spurt of a doubling of Internet trafKc every 3 or 4 months in 1995 and 1996. What were the causes of the 100-fold explosion in Internet backbone traffic over the 2-year period of 1995 and 1996? We do not have precise data, but it appears that there were four main factors, all interrelated. Browsers passed some magic threshold of usability, so many more people were willing to use computers and online information services. Users of the established online services, primarily AOL, CompuServe, and Prodigy, started using the Internet. The text-based transmissions of those services, which probably averagedonly a few hundred bits per second per connected user, were replaced by the graphicsrich content of the Web, so transmission rates increased to a few thousand bits per second. Finally, flat rate access plans led to a tripling of the time that individual users spent online [Odlyzko3], as well as faster growth in number of users. The Internetwas able to support this explosionin use because it was utilizing the existinginfrastructure of the telephone network. At that time, the Internet was tiny compared to the voice network.It is likely that the data network that handles control and billing for the AT&T long-distance voice services by itself was carrying more traffic than the NSF Internet backbone did at its peak at the end of 1994. Today, by contrast, the public Internet is rapidly moving toward being the main network, so quantum jumps in traffic cannot be tolerated so easily. In late 1999, a new application appeared that attracted extensive attention and led to many predictions that network traffic would see a major impact. It was Napster. At the time, numerous articles in the press cited Napster’s ability to “overwhelm Internet lines,” and have claimed that it has forced numerous universities to ban or limit its use. The impression one got from those press reports was that Napster was causing a quantumjump in Internet traffic, and was driving the traffic growth rates well beyond the normal range. However, upon close examination this does not appear to be completely accurate, and the use of Napster has not increased growth rates much beyond the annual doubling or tripling rates, even within university environments,where Napster is most popular. That is not to say that it has not resulted in huge amounts of traffic, nor that it has not had serious impact on several major networks. Napster provides software that enables users connected to the Internet to exchange andor download MP3 music files. The Napster Web site matches users seeking certain music files with other users who have those files on their computer. The Napster system preferentially uses machines that have high
2. Growth of the Internet
43
bandwidth connections as sources of files. This means that universities are the primary sources, since other organizations with fast dedicated links, mainly corporations, do not allow such traffic. The result is that although college students are often cited as the greatest users of MP3 files, it is the traffic from universities that gets boosted the most. (Because that direction of traffic is typically much less heavily used than the reverse one, the impact of Napster is much less severe than if the dominant direction of traffic were reversed.) Regular modem users are usually not affected, because their connections are too slow. However, the proliferation of cable modems and DSL connections that have “always-onyy high-bandwidth connectivity is leading to problems for some residential users, especially since the uplink is the one that invariably has the more limited bandwidth. A key reason that Napster is of great interest to us is that similar types of sharing applications effectively turn consumers of information into providers of information. u h e World Wide Web was designed for such information sharing, but for some types of files Napster and its kin are preferable.) These applicationswill effectively turn traditional consumerPCs into Internet servers that will output large amounts of traffic to other users. In Napster’s case this has been predominantly MP3 musicfiles, but other programs, such as Gnutella, work with more general data. It is highly probable that such applicationscould be one of the key applications that fuel the continued annual doubling or tripling of data traffic. Napster first became noticeable in the summer of 1999. Its share of the total Internet traffic on many of the university networks has grown from essentially nothing to around 25% of the total traffic by mid to late 2000. In [CoffmanO2] the traffic generated by Napster and its impact on various networks was examined. The amount of Napster traffic that is reported by several university networks (such as University of California at Santa Cruz, University of Michigan, University of Indiana, University of California at Berkeley, Northwestern University, and Oregon State University to name a few) ranges from around 20 to 50%. However, the reported numbers are often very preliminary, and in some cases they compare Napster traffic to total traffic, whereas in others it appears that the high values may represent a comparison only to the out traffic. In any event, this is a phenomenal growth rate for any single application. Since it started from zero and our data only goes out to about a year from that time, it is risky to extrapolate this initial explosion out indefinitely. In most cases [CoffmanO2], Napster has had a noticeable effect on the growth rate of traffic on this campus, but not an outlandish one. Several networks, such as that of the University of Wisconsin-Madison that report Napster traflk making up as much as 30% of the total, are not doing anything to limit Napster because they claim that they still have plenty of bandwidth. Others have imposed limits on the total bandwidth available to the dormitories.
44
Kerry G. Coffman and Andrew M. Odlyzko
Aside from Napster, occasionally even a large institution will experience a local perturbation in its data traffic patterns caused by one particular application. For example, the SETI@home distributed computing project (http://setiathome.ssl.berkeley.edu)uses idle time on about three million PCs (as of mid 2001) to search for signs of extraterrestrial intelligence in signals collected by radio telescopes. This project is run out of the Space SciencesInstitute at the University of California at Berkeley, and within a year of inception it accounted for about a third of the outgoing campus traffic [McCredie]. (Moreover, this was extremely asymmetrical traffic, with large sets of data to be analyzed going out to the participating PCs and small final results coming back. That most of the data went away from campus made this application less disruptive than it would have been otherwise.) Its disruptive effect is moderated by limiting its transmission rate to about 20 Mb/s. At the University of California at Santa Cruz, a complete copy of the available genome sequence was made available for public download in early July 2000. This, combined with coverage in the popular press and on Slashdot, led to an immediate surge in traffic, far exceeding the effects of Napster. If the interest in this database continues, it will require reengineering of the campus network. The SETI@home project is interesting for several reasons. It is cited in [McCredie] as a major new disruptive influence. Yet it contributes only about 20 Mb/s to the outgoing traffic. An increasing number of PCs and workstations are connected at 100Mb/s, and even Gigabit Ethernet (1,000 Mb/s) is coming to the desktop. This means that for the foreseeable future, a handful of workstations will, in principle, be capable of saturating any Internet link. Given the projections for bandwidth, a few thousand machines will continue to be capable of saturating all the links in the entire Internet. Thus control on user traffic will have to be exercised to prevent accidental as well as malicious disruptions of service. However, it seems likely that such control could be limited to the edges of the network. In fact, such control will pretty much have to be exercised at the edges of the network. QoS (Quality of Service) will not help by itself, since a malicious attacker who takes control of a machine will be able to subvert any automatic controls. Finally, after considering current disruptions from Napster and SETI@home, we go back and consider browsers and the Web again. They were cited as disruptive back in 1994 and 1995. (Mosaic was first released unofficially around the middle of 1993, officially in the fall of 1993, and took off in 1994.) However, when we consider the growth rates for the University of Waterloo, for MichNet [CoffmanOl], or for SWITCH (which apparently had regular growth throughout the 1990s according to [Harms]), we do not see anything anomalous, just the steady doubling of traffic each year or so. If we consider the composition of the traffic, there were major changes. For example, Fig. 2.4 shows the evolution of traffic between the University of Waterloo and the Internet. (It is based on analysis of traffic during the third week in each March, and more complete results
2. Growth of the Internet
45
are available at http://www.ist.uwaterloo.ca/cn/Stats/ext-prot.html.) The Web did take over, but much more slowly than on Internet backbones. There are no good data sets, but it has been claimed that by the end of 1994, Web traffic was more than half of the volume of the commercial backbones. On the other hand, the data for the NSFNet backbone, available at http://www.merit.edu/merit/archive/nsfnet/statistics/index.html, show that Web traffic was only approaching 20% there by the end of 1994, a level similar to that for the University of Waterloo. Thus, at well-wired academic institutions such as the University of Waterloo and others that dominated NSFNet traffic, the impact of the Web was muted. Perhaps the main lesson to be drawn from the discussion in this section is that the most disruptive factor is simply rapid growth by itself. A doubling of traffic each year is very rapid, much more rapid than in other communication services. Figure 2.4 shows e-mail and netnews shrinking as fractions of the traffic at the University of Waterloo, from a quarter to about 5%. Yet the byte volume of these two applications grew by a factor of 12 during the 6 years covered by the graph, for a growth rate of over 50% per year, which is very rapid by most standards. If we are to continue the doubling of traffic each year, new applicationswill have to keep appearing and assuming dominant roles. An interesting data point is that even at the University of Wisconsin in Madison, which analyzes its data traffic very carefully, about 40% of the transmissions escape classification.That is consistent with information from a few corporate networks, where the managers report that upwards of half of their traffic is of unknown types. (A vast majority of network managers do not even attempt to perform such analyses.) This shows how difficult coping with rapid growth is. internet traffic at the University of Waterloo
0 0
-
m
E
9
u - 0
o m
a
0, m c
p 0)
n 0
cu
1994
I
I
I
I
I
I
1995
1996
1997 year
1998
1999
2000
Fig. 2.4 Composition of traffic between the University of Waterloo and the Internet based on data collected in March of each year.
46
Kerry G. Coffman and Andrew M. Odlyzko
8. Moore’s Law for Data Traffic The approximate doubling of transmission capacity of each fiber that is described in [CoffmanO2] is analogous to the famous Moore’s Law in the semiconductorindustry. In 1965, Gordon E. Moore, then in charge of R&D at Fairchild Semiconductor,made a simple extrapolation from three data points in his company’s product history. He predicted that the number of transistors per chip would about double each year for the next 10 years. This prediction was fulfilled, but when Moore revisited the subject in 1975, he modified his projection for further progress by predicting that the doubling period would be closer to 18 months. (For the history and fuller discussion of Moore’s Law, see [Schaller].) Remarkably enough, this growth rate has been sustained over the past 25 years. There have been many predictions that progress was about to come to a screeching halt (including some recent ones), but the most that can be said is that there may have been some slight slowdown recently. (For example, according to the calculations shown in FlderingSE], the number of transistors in leading-edge microprocessors doubles every 2.2 years. On the other hand, the doubling period is lower for commodity memories.) Experts in the semiconductor area are confident that Moore’s 1975 prediction for rate of improvement can be fulfilled for at least most of the next decade. Predictions similar to Moore’s had been made before in other areas, and in [Licklider]they were made for the entire spectrum of computing and communications. However, it is Moore’s Law that has entered the vernacular as a description of the steady and predictable progress of technology that improves at an exponential rate (in the precise mathematical sense). Moore’s Law results from a complex interaction of technology, sociology, and economics. No new laws of nature had to be discovered, and there have been no dramatic breakthroughs. On the other hand, an enormous amount of research had to be carried out to overcome the numerous obstacles that were encountered.It may have been incremental research, but it required increasing ranks of very clever people to undertake it. Furthermore, huge investments in manufacturing capacity had to be made to produce the hardware. Perhaps even more important, the resulting products had to be integrated into work and lifestyles of the institutions and individuals using them. For further discussions of the genesis, operations, and prospects of Moore’s Law, see [ElderingSE, Schaller]. The key point is that Moore’s Law is not a natural law, but depends on a variety of factors. Still, it has held with remarkable regularity over many decades. Although Moore’s Law does apply to a wide variety of technologies, the actual rates of progressvary tremendouslyamong Merent areas For example, battery storage is progressing at a snail‘s pace’ compared to microprocessor improvements. This has signscant implications for mobile Internet access, limiting processor power and display quality. Display advancesare more rapid than those in power storage, but nowhere near fast enough to replace paper
2. Growth of the Internet
47
as the preferred technology for general reading, at least not at any time in the next decade. (This implies, in particular, that the bandwidth required for a single video transmission will be growing slowly.) Dynamic Random Access Memories (DRAMS) is growing in size in accordance with Moore’s Law, but their speeds are improving slowly. Microprocessors are rapidly increasing their speed and size (which allows for faster execution through parallelism and other clever techniques), but memory buses are improving slowly. For some quantitative figures on recent progress, see [Grays]. From the standpoint of a decade ago, we have had tidal waves of just about everything: processing power, main memory, disk storage, and so on. For a typical user, the details of the PC on the desktop (MHz rating of the processor, disk capacity) do not matter too much. It is generally assumed that in a couple of years a nav and much more powerful machine will be required to run the new applications, and that it will be bought for about the same price as the current one. In the meantime, the average utilization of the processor is low (since it is provided for peak performance only), compression is not used, and wasteful encodings of information (such as 200 KB Word documents conveying a simple message of a few lines) are used. The stress is not on optimizing the utilization of the PC’s resources, but on making life easy for the user. To make life easy for the end user, though, clever engineering is employed. Because the tidal waves of different technologies are advancing at different rates, optimizing user experience requires careful architectural decisions [Grays, HennessyP]. In particular, since processing power and storage capacity are growing the fastest, while communication within a PC is improving much more slowly, elaborate memory hierarchies are built. They start with magnetic hard disks and proceed through several levels of caches, invisibly to the user. The resulting architecture has several interesting implications, which are explored in [Grays]. For example, mirroring disks is becoming preferable to RAID (Redundant Arrays of Inexpensive Disks) fault-tolerant schemes that are far more efficient but slower. The density of magnetic disk storage increased at about 30% per year from 1956 to 1991, doubling every 2; years [Economist]. (Total deployed storage capacity increased faster, as the number of disks shipped grew.) In the 1990s, the growth rate accelerated, and in the late 1990sincreased yet again. By some accounts, t h densities ~ in disk drives are about doubling each year. For our purposes, the most relevant figure will be total storage of disk drives. Table 2.5 shows data from an IDC study, which shows storagecapacity shippedeach year just about doubling through the year 2000, and then slowing down. However, that study was prepared in 1998, and since then IDC has revised upwards its estimatesfor disk storage systems toward a continuation of the doubling trend. Similar projections from Disk/Trend (http://www.disktrend.com/)also suggest that the total capacity of disk drives shipped will continue doubling through at least the year 2002. Given the advances in research on magnetic storage, it seems that a doubling each year until the year 2010 might be achievable (with
48
Kerry G. Coffman and Andrew M. Odlyzko Table 2.5 Worldwide Hard Disk Drive Market (based on September 1998 and August 2000 IDC reports) Year 1995 1996 1997 1998 1999 2000 200 1 2002 2003 2004
Revenues (bizfions)
Storage Capacity (terabytes)
$21.593 24.655 27.339 26.969 29.143 32.519 36.219 40.683
76,243 147,200 334,791 695,140 1,463,109 3,222,153 7,239,972 15,424,824 30,239,756 56,558,700
some contributionfrom higher revenues, as shown in Table 2.5, but most coming from better technology).After about 2010, it appears that magnetic storage progress will face serious limits, but by then more exotic storage technologies may become competitive. It seems safest to assume that total magnetic disk storage capacity will be doubling each year for the next decade. However, even if there is a slowdown, say to a 70% annual growth rate, this will not affect our arguments too much. The key point is that storage capacity is likely to grow at rates not much slower than those of network capacity. Furthermore, total installed storage is already immense. Table 2.5 shows that at the beginning of the year 2000, there were about 3,000,000TB of magnetic disk storage. If we compare that with the estimates of Table 2.1 for network traffic, we see that it would take between 250 and 400 months to transmit all the bits on existing disks over the Internet backbones. This comparison is meant as just a thought exercise. The backbones considered in Table 2.1 arejust those in the United States, whereas disks counted in Table 2.5 are spread around the world. A large fraction of the disk space is spare, and much of the content is duplicated (such as those hundreds of millions of copies of Windows 98), so nobody would want to send them over the Internet. Still, this thought exercise is useful in showing that there is a huge amount of digital data that could potentially be sent over the Internet. Further, this pool of digital data is about doubling each year. An interesting estimate of the volume of information in the world is presented in [Lesk]. It shows that already in the year 1997we were on the threshold of being able to store all data that has ever been generated (meaning books, movies, music, and so on) in digital format on hard disks. By now we are well
2. Growth of the Internet
49
past that threshold, so future growth in disk capacities will have to be devoted to other types of data that we have not dealt with before. Some of that capacity will surely be devoted to duplicate storage (such as a separate copy of an increasingly bloated operating system on each machine). Most of the storage, though, will have to be filled by new types of data. The same process that is yielding faster processors and larger memories is also leading to improved cameras and sensors These will yield huge amounts of new data that have not been available before. It appears impossible to predict precisely what type of data this will be. Much is likely to be video storage from cameras set up as security measures or ones that record our every movement. There could also be huge amounts of data from medical sensors on our bodies. What is clear, though, is that “[tlhe typical piece of information will never be looked at by a human being” [Lesk]. There will simply not be enough of the traditional “content” (books, movies, music) nor even enough of the less formal type of “content” that individuals will be generating on their own. Huge amounts of data that is machine generated for machine use suggests that data networks will also be dominated by transfers of such data. This was already predicted in [deSolaPITH],and more recently in [Odlyzko2,StArnaud, StArnaudCFM].Given an exponential growthrate in volume of data transfers, it was clear that at some point in the future most of the data flying through the networks would be neither seen nor heard by any human being. Thus, we can expect that streaming media with real-time quality requirements will be a decreasing fraction of total traffic at some point within the next decade. There will surely be an increase in the raw volume of streaming real-time traffic, as applications such as videoconferencing move onto the Internet. However, as a fraction of total trafiic, such transmissionswill not only decrease eventually, but may not grow much at all even in the intermediate future. (Recall that at the University of Waterloo over the last 6 years, the volume of e-mail grew about 50% a year, but as a fraction of total trafiic it is almost negligible now.) The huge imbalance in volume of storage and capacities of long-distance data networks means that even the majority of traditional “content” will be transmitted as files, and not in streaming form. For more detailed arguments supporting this prediction, see [Odlyzko2]. This development, in which “content”is sent around as files for local storage and playback, is already making its appearance with MP3, Napster, and related programs. The huge hard disk storagevolumes also mean that most data will have to be generated locally. There will surely also be much duplication (such as operating systems, movies, and so on that would be stored on millions of computers). Aside from that, there will surely be huge volumes of locally generated data (e.g., from security cameras and medical sensors) that will be used (if at all) only in highly digested form. The examples in [Coffman02] support the notion that there is a “Moore’s Law” for data traffic, with transmission volumes doubling each year. Even at large institutions that already have access to state-of-the art technology,
50
Kerry G. Cofitnan and Andrew M. Odlyzko
data traffic to the public Internet tends to follow this rule of doubling each year. This is not a natural law, but, like all other versions of Moore’s Law, reflects a complicated process, the interaction of technology and the speed with which new technologies are absorbed. A Moore’s Law for data traffic is different from those in other areas, since it depends in a much more direct way on user behavior. In semiconductors, consumer willingness to pay drives the research, development, and investment decisions of the industry, but the effects are indirect. In data traffic, though, changes can potentially be much faster. A residential customer with dial-up modem access to the Internet could increase the volume of data transfer by a factor of about five very quickly. All it would take would be the installation of one of the software packages that prefetch Web sites that are of potentialinterest and that fill in the slack between transmissions initiated by the user. Similarly, a university’s T3 connection to the Internet could potentially be filled by a single workstation sending data to another institution. Thus any Moore’s Law for data traflic is by nature much more fragile than the standard Moore’s Law for semiconductors,for example. Thus it is remarkable that we see so much regularity in growth rates of data transfers. Links to the public Internet are usually the most expensive parts of a network, and are regarded as key choke points They are where congestion is seen most frequently at institutional networks. Yet the “mere” annual doubling of data traffic even at institutions that have plenty of spare capacity on their Internet links means that there are other barriers that matter. The obvious one is the public Internet itself. It is often (some would say usually) congested. A terabit pipe does not help if it is hooked up to a megabit link, and so providing a lightly utilized link to the Internet does not guarantee good end-to-end performance. Yet that is not the entire explanation either, since corporate Intranets, which tend to have adequate bandwidth and seldom run into congestion, tend to grow no faster than a doubling of traflic each year. There are other obstructions, such as servers, middleware, and, perhaps most important, services and user interfaces People do not care about getting many bits. What they care about are the applications. However, applications take time to be developed, deployed, and adopted. To quote J. Licklider (who probably deserves to be called “the grandfather of the Internet” for his role in setting up the research program that led to the Internet’s creation): A modern maxim says: “People tend to overestimate what can be done in one year and to underestimate what can be done in five or ten years.” Picklider]
“Internet time,” where everything changes in 18 months, has a grain of truth, but is largely a myth. Except for the ascendancy of browsers, most substantial changes take 5 to 10 years. As an example, it has been at least 4 years since voice over IP was first acclaimed as the ‘‘next big thing.” Yet its impact so far has been surprisinglymodest. It is coming, but it is not here today, and it won’t be here tomorrow. People take time to absorb new technologies.
2. Growth o f the Internet
51
What is perhaps most remarkable is that even at institutions with congested links to the Internet, traffic doubles or almost doubles each year. Users appear to find the Internet attractive enough that they exert pressure on their administration to increase the capacity of the connection. Existing constraints, such as those on e-mail attachments, or on packetized voice or video, as well as the basic constraint of limited bandwidth, are gradually loosened. Note that this is similar to the process that produces the standard Moore’s Law for PCs. Intel, Micron, Toshiba, and the rest of the computer industry would surely produce faster advances if users bought new PCs every year. Instead, a typical PC is used for 3 to 4 years. On one hand there is pressure to keep expenditures on new equipment and software under control, and also to minimize the complexity of the computing and communicationssupportjob. On the other hand, there is pressure to upgrade, either to better support existing applications or to introduce new ones. Over the last three decades, the conflict between these two pressures has produced a steady progress in computers. Similar pressures appear to be in operation in data networking. In conclusion, we cannot be certain that Internet t r a c will continue doubling each year. All we can say is that historically it has tended to double each year. Still, trends in both transmission and in other information technologies appear to provide both the demand and the supply that will allow a continuing doubling each year. Since betting against such Moore’s laws in other areas has been a loser’s game for the last few decades, it appears safest to assume that data traffic will indeed follow the same pattern, and grow at close to 100% per year.
9. Further Economic and Technical Considerations A frequently asked question concerns the elasticity of demand for data transmission capacity. However, for long-range projections it might be more useful to think of analogies with the computer industry. In that industry, product managers clearly do think about elasticities in the short or intermediate terms. From a long-range perspective, though, what dominates are the effects of Moore’s Law. Table 2.6 (drawn from [FishburnO]) shows a dozen years from the history of Intel. The leading microprocessor sold for roughly a constant price all during this period. However, its power was increasing at the exponential rate given by Moore’s Law. Intel’s total revenues (and profits) grew, as more processors were being sold, but this growth rate was considerably more modest than that of the computing power. Users found the increasing computational power of new PCs sufficiently attractive that they not only bought new PCs, but increased their total spending. They did this even though most of that power was sitting idle, and it was only the occasional bursts of recomputing a spreadsheet or bringing up a presentation package that mattered. A similar
52
Kerry G. Coffman and Andrew M. Odlyzko
Table 2.6 Intel and Its Microprocessors(each year lists the most powerful General Purpose Microprocessors Sold by Intel, Its Computing Power, Price at the End of the Year (in Dollars), and Intel’s Revenues and Profits for That Year (in Millions of Dollars)) Price (dollam)
Year
Processor
Mips
1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997
386 DX (16MHz) 386 DX (20 MHz) 386 DX (25 MHz) 486 DX (25 MHz) 486 DX (33 MHz) 486 DX (50 MHz) DX2 (66 MHz) Pentium (66 MHz) Pentium (100MHz) Pentium Pro (200 MHz)
5 6 8 20 27 41 54 112 166 400
950 950 644 600 898 935 1,325
Pentium I1 (300 MHz)
600
735
300
Revenue (millions of dollars)
Net Profit (millions of dollars)
1,265 1,907 2,875 3,127 3,922 4,779 5,844 8,782 11,521 16,202 20,847 25,070
- 173 248 453 391 650 819 1,067 2,295 2,266 3,566 5,157 8,945
evolution might take place in networking. Total spendingmay (subjectto business cycles) increase at a moderate pace, while the bandwidth and traffic grow at rates determined by technological progress. If that happens, we are likely to see traffic and capacity about doubling each year, with capacity growth faster than that of traffic.
10. Conclusions Much of the almost hyperactivitywithin the optical fiber telecommunications industry over the past few years can be traced to the perceived and real growth of the traffic on the Internet. We maintain that the overall growth rate of the Internet for most of its existence (despite some excursions) was remarkably close to “doubling every year,” and we anticipate that this rate will continue into the foreseeable future. In effect, we see a type of Moore’s Law associated with the growth of data traffic. This type of growth rate is in sharp contrast to the historical growth rates of various methods of communications (including conventional mail, telegraph service, and traditional voice phone service) that tended to be no greater (and typically much less) than about 10Y0per year. Still, even though a doubling each year represents very fast growth, it is only comparable to the rate of progress in transmission capacity. Hence we are
2. Growth of the Internet
53
unlikely to see the huge increases in spending on optical communication that many business plans had been based on. Throughout the history of the Internet there have only been two “killer applications”: e-mail and the Web (including Web browsers). Several events conspired that allowed an unprecedentedexplosion (roughly 100-fold increase) in Internet traffic in the 1995-1996 time frame, and the Internet was able to handle this because it made use of the existing telephone industry infrastructure. Because the Internet is quickly approaching the point at which it is the predominant network, it is very unlikely that such huge growth rates could be so easily supported in the future. It also appears that, aside from short-range perturbations, there will be neither a “bandwidth glut” nor a “bandwidth shortage” in the foreseeable future, in that supply and demand will be growing at comparable rates. As such, it is very likely that pricing will begin to play an even more important role in the evolution of traffic. Throughout most of the 1990s, data transmission prices were increasing. However, there are recent signs that they are beginning to decrease, and in some cases, especially across the Atlantic and on major transcontinental routes in the United States, they have decreased dramatically. If they begin to decrease rapidly in general, then many of the constraints on usage that exist today may very likely start to ease. We are likely to see capacity growing somewhat faster than traffic, a continuation of the trend we have already seen in the last few years. We also believe that “file” transfers, and not real-time streaming,will remain dominant on the network. Streaming real-time transmissionswill undoubtedly grow in absolute terms, and as a fraction of the total traffic it may increase for a while. However, in all likelihood it will eventually begin to decline as the demand for this type of traffic will not grow as fast as network capacity. We foresee sharing applications as a likely candidate to fuel traffic growth. One of the first major examples of this was Napster, because it effectively turned consumers of information into providers of information. It is extremely likely that such file sharing applications will be some of the key applications that continue to fuel the annual doubling of data traffic.
References [Abbate] [Baran] [Boardwatch] [Bruno]
J. Abbate, Inventing the Internet, MIT Press, 1999.
I? Baran, On distributed communications network, IEEE Trans. Comm. Systems, vol. 12, 1964, pp. 1-9. Boardwatch. Available at http://www.boardwatch.com. L. Bruno, Fiber optimism: Nortel, Lucent, and Cisco are battling to win the high-stakes fiber-optics game, Red Herring, June 2000. Available at http://www.herring.com/mag/issue79/mag-fiber79.html.
54
Kerry G. Coffman and Andrew M. Odlyzko
1Cerf-I [CerfKI
[Cochrane]
[CoffmanOl]
[Coffman02]
[CTIA]
[Cyberspace]
[deSolaPITH]
PWLI
[Economist] [ElderingSE]
[Fishburno]
FlOYdPI [Galbi]
V. G. Cerf, A brief history of the Internet and related networks. Available at http://www.isoc.org/internethistory/cerf html. V. G. Cerf and R. E. Kahn, A protocol for packet network interconnection, IEEE Trans. Comm. Tech., vol. COM-22, May 1974, pp. 627-641. N. Cochrane, We’re insatiable: Now it’s 20 million million bytes a day, Melbourne Age, January 15, 2001. Available at http:llwww .it.fairfax.com.au/networking/200 10115/A13694-2001Jan15.html. K. G. Coffman and A. M. Odlyzko, The size and growth rate of the Internet, First Monday, October 1998. Available at http://fkstmonday.org/. Also available at http://www.dtc.umn.edu/ -0dlyzko. K. G. Coffman and A. M. Odlyzko, Internet growth: Is there a “Moore’s Law” for data traffic?, Handbook of Massive Data Sets, J. Abello, P. M. Pardalos, and M. G. C. Resende, eds, Kluwer, 200 1, in press. Available at http://www.dtc.umn.edu/-odlyzko. CTIA (Cellular Telecommunications Industry Association), SemiAnnual FEreleSs Industry Survey, June 1985 to June 2000. Available at http:l/www.wow-com.codwirelesssurvey/. Geography of Cyberspace Directoiy: Internet Trafic and Demographic Statistics. Available at http://www.cybergeography.org/ statistics.html. I. de Sola Pool, H. Inose, N. Takasaki, and R. Hurwitz, Communications Flows: A Census in the Unitedstates andJapan, North-Holland, 1984. D. A. Dunn and A. J. Lipinski, Economic considerations in computer-communication systems, pp. 371- 422 in ComputerCommunication Networh, N. Abramson and F. F. Kuo, eda, Prentice-Hall, 1973. Not Moore’s Law, The Economist, July 12,1997. C . A. Eldering, M. L. Sylla, and J. A. Eisenach, Is there a Moore’s Law for bandwidth? IEEE Communications Magazine, October 1999, pp. 2-7. I? C . Fishburn and A. M. Odlyzko, Dynamic behavior of differential pricing and Quality of Service options for the Internet, pp. 128-139 in Proc. First Intern. Con$ on Information and Computation Economies (ICE-98), ACM Press, 1998. Available at http://www.dtc.umn.edu/-odlyzko. S. Floyd and V. Paxson, Dimculties in simulating the Internet, IEEE/ACM Damactions on Networking, in press. Available at ht tp://www.aciri.orglfloydpapers.html. D. Galbi, Bandwidth use and pricing trends in the U.S., Telecommunications Policy, vol. 24, no. 11, December 2000. Available at http://www.galbithink.org.
2. Growth of the Internet
55
J. Gray and P. Shenoy, Rules of thumb in data engineering,
[Green]
[Harms]
[HennessyP] [Hobbes] [Houghl [Kleinrockl]
Proc. 2000 ZEEE Intern. Con$ Data Engineering. Available at http://research.microsoft.com/-gray. E. E. Green, Communicationsspectra by the wholesage2012 A.D., Proc. IRE, vol. 50,1962, pp. 585-587. Reprinted in Proc. ZEEE, vol.
87,1999,1293-1295. J. Harms, From SWITCH to SWITCH*-extrapolating from a case study, Proc. INET '94, pp. 341-1 to 341-6. Available at http:l/ info.isoc.org/iso~whatis/conferences/ine~94lpapers/index. html. J. L. Hennessy and D. A. Patterson, Computer Architecture: A Quantitative Approach, Morgan Kaufmann, 1990. Hobbes Internet Emeline. Available at http:/lwww.zakon.org/ robert/internet/timeline/. R. W. Hough, Future data traffic volume, ZEEE Computer, September-October 1970, pp. 6-12. L. Kleinrock, Information flow in large communicationsnetworks, RLE Quarterly Progress Report, July 1961, pp. 1-35. Available at
http://www.lk.cs.ucla.edu/LK/Bib/REPORT/PhD. [Kleinrock2] [Kleinrock3]
L. Kleinrock, Communication Nets; Stochastic Message Flow and Delay, McGraw-Hill, 1964. L. Kleinrock, ISDN-The path to broadband networks, Proc. ZEEE, V O ~ .79,
[Leiner]
[Lesk] [Licklider] [LickC] [Lucky11 [Lucky21 [McCredie]
[MeekerMJ]
1991, pp. 112-117. B. M. Leiner, V. G. Cerf, D. D. Clark, R. E. Kahn, L. Kleinrock, D. C. Lynch, J. Postel, L. G. Roberts, and S. Wolf€, A Brief History of the Internet, Version 3.31, August 4, 2000. Available at http://www.isoc.org/internetlhistory/brief.html. M. Lesk, How much information is there in the world? 1997, unpublished paper. Available at http://www.lesk.com/mlesk/diglib.html. J. C. R. Licklider, Libraries of the Future, MIT Press, 1965. J. C . R. Licklider and W. Clark, On-Line Man Computer Communications, August 1962. R. W. Lucky, New communications services-What does society want? Proc. ZEEE, vol. 85, 1997, pp. 15361543. R. W. Lucky, Through a glass darkly-Viewing communicationsin 2012 from 1961, Proc. IEEE, vol. 87, 1999, pp. 1296-1300. J. McCredie, UC Berkeley must manage campus network growth, The Daily Californian, March 14, 2000. Available at http://www.dailycal.org/article.asp?id=1912&ref=news M. Meeker, M. Mahaney, and D. Joseph, The Internet userlusage ecosystem framework, Morgan Stanley Dean Witter report, January 24, 200 1. Available at http://www.msdw.com/techresearch/ index.htm1. The Multi-Router Traflc Grapher of Tobias Oetiker and Dave Rand,
information and links to sites using it at http://ee-staff.ethz.ch/ -oetiker/webtools/mrtg/mrtg.html.
56
Kerry G. Coffman and Andrew M. Odlyzko
U. S. Information Infrastructure Task Force, The NationalZnformation Znffastructure.Available at http://www.ibiblio.orglniiltoc.html. [Nolll] A. M. Noll, Znhoduction to Telephones and Telephone Traflc, 2nd ed., Artech House, 1991. A. M. Noll, Highway of Dreams: A Critical Appraisal of the Commu[No1121 nications Superhighway, Lawrence Erlbaum Associates, 1997. pol131 A. M. Noll, Does data traffic exceed voice traffic? Comm. ACM, June 1999, pp. 121-124. Nua Internet Surveys. Available at http://www.nua.com. [Nual A. M. Odlyzko, Data networks are lightly utilized, and will stay [OdlYZko11 that way. Available at http://m.dtc.umn.edu/-odlyzko. [Odlyzko2] A. M. Odlyzko, The history of communications and its implications for the Intenet. Available at http://www.dtc.umn.edu/-odlyzko. [Odlyzko3] A. M. Odlyzko, Internet pricing and the history of communications, Computer Networh, vol. 36, 2001, pp. 493-517. Also available at http://www.dtc.umn.edu/-odlyzko. J. A. Polly, Surfing the Internet: An introduction, Wilson Library Bulletin, June 1992, pp. 38-42. Available at http://www.netmom .com/about/surfing.shtml. [ReichlLS] P.Reichl, S. Leinen, and B. Stiller, A practical review of pricing and cost recovery for Internet services, Proc. 2nd Internet Economics Worhhop Berlin (IEW '99), Berlin, Germany, May 28-29, 1999. Available at http://www.tik.ee.ethz.ch/-cati/. R. R. Schaller, Moore's law: Past, present, and future, IEEE Spec[Schaller] trum, vol. 34: no. 6, June 1997, pp. 52-59. Available through Spectrum online search at http://www.spectrum.ieee.org. [StAmaud] B. St. Arnaud, The future of the Internet is NOT multimedia, Network World, November 1997. Available at http://www.canarie.ca/ -bstarn/publications. html . [StAmaudCFM] B. St. Arnaud, J. Coulter, J. Fitchett, and S. Mokbel, Architectural and engineering issues for building an optical Internet. Short version in Proc. SOC.Optical Engineering, 1998. Full version available at http://www.canet3.net. [Standage] T. Standage, n e Kctorian Internet: n e Remarkable Story of the Telegraph and the Nineteenth Century5. On-line Pioneers, Walker, 1998. S. Taggart, Telstra: The prices fight, Wired News, http://www.wired .com/news/politics/O,1283,32961,OO.html. €? M. Walker and S. L. Mathison, Regulatory policy and future data transmission services, pp. 295-370 in ComputerCommunication Networh, N. Abramson and F. E Kuo, eds, Prentice-Hall, 1973. W. W. Wu and A. Livne, ISDN A snapshot, Proc. ZEEE, vol. 79, 1 9 9 1 , ~103-111. ~. PI11
Chapter 3
Optical Network Architecture Evolution
John Strand AT&T Laboratories, Middletown, New Jersey
1. Introduction An Optical Transport Network (OTN) is composed of interconnectednetwork elements (NEs), plus software and operational processes that must function together to provide services. Ongoing advances in technology will cause significant changes in OTN architecture; however, equally important will be the growth and evolution of the services it transports, particularly the Internet. In addition, business changesin the telecommunicationsindustry will be very critical. This chapter tries to weave the technology, services, and business stories together to indicate how they are shaping the architecture of the emerging optical network. Because most of the readers of these volumes are primarily technologists, tutorial material has been included. 1.1. WHAT IS AN OPTICAL TRANSPORTNETWORK? The very definition of “Optical Transport Network” illustrates why these stones are interrelated. All would agree that optical fiber will be the physical layer of an OTN. There is less agreement on whether a network making extensive use of electronics for regeneration and switching should be called an OTN. Most of the audience for this volume are presumably most interested in all-optical OTNs; however, for a variety of business and service-related reasons that will be discussed later, many of the first generation “optical crossconnects” perform electronics-based functions like multiplexing DS-3s and slower speed SONET OC-ns into OC-48s and OC-192s. From this perspective, SONET is an “opaque one-wavelength”optical network, as Green [47]pointed out, even though much of its characteristic functionality is implemented in electronics. We will take a “broad church” approach to defining an OTN. Given the nature of this volume, we will concentrate primarily, when possible, on networks built from optical components; however, when necessary we will use the term to include networks transporting STS-1 (52 Mb/s) and larger connections that have an optical physical layer. When necessary to be precise, we will use the term “photonic” instead of “optical” to indicate that an all-optical The views expressed are those of the author and not necessarily those of AT&T.
57 OPTICAL FIBER TELECOMMUNICATIONS, VOLUME IVB
Copyright 0 2002, Elsevier Science (USA). AU rights of reproduction in any form reserved. ISBN: 0-12-395173-9
58
John Strand
situation is being discussed. Thus, a “PhotonicTransport Network” (PTN) will be an all-optical OTN, and “Photonic Cross-Connect” (PXC) and “Photonic Add/Drop Multiplexer” (PADM) will be used when all-optical equipment is under discussion. We will call a communications channel through an OTN a “circuit.” It is frequentlycalled a “wavelength”or a “lightpath,”but because of the possibility that a connection might be partially electronic,we will stick with “connection.” If it is all-optical,we will also use the term “optical channel” (OCh). 1.2. SCOPE OF THIS CHAPTER
Because other chapters in the current volume deal with submarine systems, metropolitan OTNs, and access OTNs, we will focus primarily on intercity networks. Because of the background of the author, most of the discussion will have a strong U.S.flavor: For example, I willuse SONET rather than SDH terminology, and I will frequently discuss issues of most interest to networks with diameters of thousands of kilometers, even though such large networks are only relevant to a handful of countries. To keep the material to be covered within bounds, only technology likely to be commercially available within the next few years is considered. Many interesting areas, such as optical packet switching, are therefore omitted. The reader should keep in mind the large error bars surrounding this whole enterprise. As an example, the architecture of actual OTNs even a few years in the future depends profoundly on the Internet: If its growth were to slow, the rate of introduction of new architectures would certainly slow, and the functionality desired (and hence the underlying technologies)might well change.
2. Technology Advances and Trends The important underlying optical technologiesare for the most part discussed elsewhere in this volume; the reader interested in the technologies per se should turn to the relevant sections. Our purpose here is twofold: (1) identify and define at a system level the building blocks on which the OTN architecture rests, and (2) point out major system-leveltrends we need to deal with in our later architecture development. 2.1. SONETBDH REFRESHER
SONET/SDHis important for optical networking for several reasons: (1) The overwhelming proportion of connections in long-haul OTNs are SONET or SDH formatted, and (2) many aspects of OTN architectures have been modeled on SONET/SDH concepts. This section gives a very brief overview of a few key aspects of these protocols that we will need later. For more complete overviews, see [18,72,73].
3. Optical Network Architecture Evolution
59
Table 3.1 Selected SONET Signal Rates SONET Name STS-1 STS-3 STS-12 STS-48 STS-192 STS-768
Name When Transported Optically
SDH Name
Signal Rate (Mbhec)
oc-1 OC-3 oc-12 OC-48 OC- 192 OC-768
STM-1 STM-4 STM-16 STM-64 STM-256
51.84 155.52 622.08 2,488.32 9,953.80 39,813.12
User Rate’ (IMb/sec) 49.54 148.61 594.43 2,377.73 9,510.91 38,043.65
’
The “User Rate” is the bandwidth actually available for user data. The difference between it and the “Signal Rate” is due to overhead information as discussed in the text.
Synchronous Optical NETwork (SONET) is a North American standard for networking developed in the mid-1980s primarily by Bellcore and standardized by ANSI [79,99, 1001. It defines the interface between two SONET network elements (NEs). More specifically, it defines a digital hierarchy of synchronous signals, including their formats and mappings of asynchronous signals (e.g., DS-1, DS-3) into these formats, and defines the electrical and optical characteristics of the interface. The Synchronous Digital Hierarchy (SDH) is a closely related standard developed by the ITU [78]. The basic SONET entity is the Synchronous Transport Signal-1 (STS-1). It operates at 51.84Mb/secondYof which 49.5 Mb/sec is usable payload and the rest overhead. The STS-1 frame structure is byte-oriented and has 9 rows and 90 columns, 4 of which are used for overhead purposes. The frame rate is 8000/second (125 ps/frame). Normally all SONET signals are bidirectional and run at the same rates in each direction. An STS-N signal (n > 1) is formed by byte-interleavingn STS-1s together. When an STS-N is transported electrically, it is called an “EC-n”; when transported optically, it is an c‘OC-n.y’ (EC stands for “Electrical Carrier,” OC for “Optical Carrier.”) SDH is virtually identical to SONET, except that its base frame is three times larger, corresponding to a SONET STS-3. It is called “STM-1,” where STM stands for “Synchronous Transfer Module.” The key SONET signal rates are summarized in Table 3.1. In their simplest form, the three basic SONET network elements are shown in Fig. 3.1. The Digital Cross-Connect System (DCS) is the most general of these NEs. In its simplest form, it has a fabric capable of cross-connecting STS-M and STS-N line-side2ports. Normally N is greater than the M , but it can be the
“Line-side” refers to the ports that are closest to the interoffice fibers; “drop-side” refers to those closest to the customer.
60
Johnstrand STS-N
I I
k
-t
STS-N
STS-N
E S
STS-M Fabric (M<=N)
s - E P
I
S T
llloopll STS-M Ports N:M Digital Cross-Connect System (DW
A
STS-M Ports N:M AddlDrop Multiplexer (ADMI
STS-M Ports N:M Multiplexer (“End Terminal”)
Fig. 3.1 Generic SONET network elements.
same. If N > M , an incoming STS-N is first demultiplexed into a number (N/M at most) of STS-Ms, which are switched through the STS-M fabric and reassembled into STS-N if they are continuing on or are dropped in the office as STS-Ms through the ports shown at the bottom. There could be many (thousands) of STS-N and STS-M ports on a single DCS. The Add-Drop Multiplexer (ADM), shown in the middle of Fig. 3.1, is a special case of a DCS. It has only two pair of bidirectional line-side ports, which are often designated “east” and “west.” Each pair has a service STS-N (“S” in the figure), and also a protection STS-N (“P”)that is normally in standby mode until needed to recover from a failure. The protection capacity may also be used for “extra traffic”--connections that will be preempted in the event of a failure. A basic multiplexer (see the right side of Fig. 3.1; often called an “end terminal”) is a further specialization. There is either a single line-side STS-N port (as shown) or a service/protection pair. It is used to multiplex a number of lower-speed signals into a single STS-N for transport through the network. All three of these types of NEs are very widely deployed today and continue to be deployed in large volumes (billions of dollars per year in the United States alone). In intercity networks, a typical ADM installed today would likely have STS-48 or STS-192 line-side ports and a mix of STS-3 and STS-12 drop-side ports. A number of specialized deployment configurations are also specified by the SONETEDH standards. The most important configuration is the Self-Healing Ring (SHR). One such SONET SHR configuration is called a “line-switched ring” for reasons that will become apparent shortly. In this configuration, up to 16 SONET ADMs are configured in a ring topology, as shown at the left in Fig. 3.2.
3. Optical Network Architecture Evolution
P=‘
!:?
P O ................
61
P
@I--I
S
0
S
ADM At Office F
Fig. 3.2 SONET line-switched ring example.
The ADMs are labeled “A” through “F,” with the service and protection STS-Ns interconnecting them labeled “S” and “P,” respectively. A connection is shown entering at A, then passing through F and E before exiting the ring at D. (Solid line is also labeled “1.”) The right side shows this connection passing through the fabric from one service port to the other. If the service connection between F and E fails, ADMs F and E would cooperate to reroute the connection over their protection connection. (Dashed line in the figure, also labeled “2.”) If there is a route failure that affects both S and P between F and E, the connection is rerouted the opposite way around the ring. (Dotted line, labeled “3.”) In both cases, the initial routing of the connection is reestablished at E. These recovery mechanisms are triggered by standardized signaling messages between F and E that are carried in the SONET overhead bytes mentioned earlier. The rerouting is done by the ADM switch fabrics as illustrated at the right in Fig. 3.2. The specialized structure of the configuration allows very rapid reaction (50 ms on rings under 1200km in circumference and with no extra traffic [79]. Sometimes 150-200 ms is used as a bound for an arbitrary ring.) In some cases the rigid restoration discipline generates convoluted restoration paths. For example, if for some reason an A-B connection was routed the long way around the ring (A-F-E-D-C-B) and E-F failed, then the restored connection would be routed A-F-A-B-C-D-E-D-C-B, even though A-B would have restored the failed connection! This is done to keep the protocol and implementations more manageable. The protection capacity is shared in this type of ring. This means that if a different link failed, the same protection capacity would be used for the connections affected. For this reason they are also called “Shared Protection Rings” (SPRING). Another important type of SONET ring is called a “path-switched ring,” for reasons discussed below. In this type of ring each connection has dedicated protection capacity. In Fig. 3.2, if the A-F-E-D connection were protected in this manner, there would have been dedicated protection capacity for the
John Strand
62
connection that would be routed A-B-C-D. The ADMs at A and D (the entry and exit points) would monitor the health of the connection and switch the connection to the A-B-C-D path if necessary. For this type of ring there is no need to know exactly where the failure occurred, because the switch is between entry and exit points. A path-switched ring is closely related to what is called “1 1 protection.” As in the path-switched ring, the connection has two dedicated paths, but in the 1 1 case, copies of the signal are continuously sent along both paths and the better quality signal is selected (at D in our example for the A to D signal). This is extremely fast and has the added advantage that no signaling between nodes is required. The drawback of 1 + 1 protection is that no extra preemptible traffic is possible on the protection capacity. SONET functionality is divided into three layers, not all of which need to be implemented by every SONET NE. The layers and their principal functions are:
+
+
0
0
0
0
Path. Maps specific services into a SONET payload; end-to-end error
and status monitoring; path protection switching. Adds path overhead. Line. Multiplexing multiple paths into a STS-N; synchronization;error and status monitoring; line protection switching. Adds line overhead. Section. Framing, scrambling, other functions associated with the preparation for physical transport. Adds section overhead. Physical. Electrical-to-Opticalconversion; actual optical transmission.
Normally an ADM would be a line and section terminating NE, whereas a regenerator would terminate only the section. A path can ride on multiple lines in series, a line on multiple sections. See Fig. 3.3. Here, PTE, LTE, and STE stand for path, line, and section terminating equipment, respectively. SONET is a byte-structured protocol. An STS-1 frame is transmitted as a string of 810 bytes that are divided into 9 “rows” of 90 bytes each. The frame is
Fig. 3.3 SONET layering.
3. Optical Network Architecture Evolution
63
divided into payload and overhead sections. The resulting frame is illustrated in Fig. 3.4. POH stands for “payload overhead.” Each of the overhead areas contain parity information to allow error checking and data communication channels to allow peer NEs to communicate. In addition, the section overhead contains framing bytes to allow frame alignment to be verified, the line overhead contains fields to control protection switching, and the path overhead identifies the type of payload. Four columns are dedicated to overhead and 86 to payload. We mentioned earlier that STS-Ns are formed by byte interleaving N STS-1s. This fragments the payload into N pieces, which is undesirable for data communications (as discussed later). To deal with this, a “concatenated” frame is also deked. Denoted STS-Nc (e.g., STS-48c or 0 C - 4 8 ~ it) ~ has one large payload rather than N smaller ones. However, it still has 3N columns of overhead, which are mostly unused. SDH is virtually identical functionally to SONET,but unfortunately the two standards use different terminology. Table 3.2 gives their correspondences.
Overhead Section p
0 Line Overhead
,,
1-
A
9
Payload
Rows
i Fig. 3.4 STS-1 frame structure.
Table 3.2 SONET and SDH TerminologyRelationship SONET Term
SDH Term
STS-N Path Line Section Line-switchedring Path-switched ring
STM-N/3 Transmission path Multiplex section Regenerator section Shared-protectionring (MSEPRING) Dedicated-protectionring (PaWDPRING)
64
John Strand
Frequency registered transmitters
Receivers
Fig. 3.5 Basic Optical Transport System (OTrS).
2.1.1. Multivendor Interworking One of the goals of SONET and SDH was to define standards that would let equipment from multiple vendors interwork. This has been achieved in simple point-to-point configurations, but in more complex configurations, such as shared protection rings, progress has been frustratingly slow. Hardware interoperability has by and large not been a serious problem; instead, software interworking has been limiting, particularly the exchange of state information between the ADMs and the handling of “operations, administration, and maintenance” functions. As a result, transport people tend to be suspicious of proposals for multivendor “mid-span meets.”
2.2. OPTICAL TRANSPORT SYSTEMS A basic Optical Transport System (OTrS) is shown in Fig. 3.5. In its most basic form, it consists of an optical multiplexer and demultiplexer and a number of optical amplifiers (OAs) between them.3 OAs in terrestrial systems usually have a nominal spacing (span length) of about 25dB (roughly 80km after allowance for splices, etc., and assuming a nominal span loss of 0.25 dB/km). Wider spacings can significantly reduce the first costs of a route, but cause difficulties for 10 Gb/sec and faster connections and also impose limits on the number of wavelengths that can be supported, therefore most operators appear to be sticking with 80 km or shorter spacings. Impairments force regeneration after about 5-7 spans (400-560 km). This is a wavelength rather than an OTrS constraint, so that if a wavelength traversed two such systems in series without regeneration between the systems, the span limit would apply to the sum of the spans on the systems.
2.2.1. Capacity Trends OTrS capacity has been increasing very rapidly. In fact, by some estimates the rate of increase is considerably faster than Moore’s Law for electronics The system shown (and the underlying technology, usually) is unidirectional; however, they are normally deployed in pairs so as to support bidirectional SONET/SDH circuits.
3. Optical Network Architecture Evolution
65
Gbls
1996
1998
2000
2002
2004
Fig. 3.6 Capacity of a single fiber (derived from data in [SI).
(see [54]). The underlying trends are analyzed in [55], from which Fig. 3.6 is derived. The years given are for commercial deployment (actual and projected). Over the 10-year period shown, capacity doubles approximately eight times due to increasesin the bits per channel, reductions in wavelength spacing, and increases in the total usable fiber bandwidth. In a recent talk [104], Rick Barry of Sycamore used bandwidth times distance as a relevant metric. He traced the evolution of capacity from 120-160 Terabits/sec*km with conventional systems (e.g., 80 OC-48s over 600km or 40 OC-192s over 400km) to today's 160&2400Tb/sec*km (e.g., 80 OC-192 over 3000km) and projected a next generation providing 3200-4800 Tb/sec*km. Important architectural implications from these trends: 0
0
0
2.2.2.
The increase in wavelength counts make the introduction of some form of mechanized cross-connect a necessity if operations costs and complexity are to be kept under control. Total OTrS costs are rising significantly more slowly than capacity. Hence unit cost ($/Gigabit) is declining. The combination of these two trends-rapidly rising capacity and declining unit costs-have formed a synergistic relationship with the rapid growth of the Internet. Internet growth has allowed the new technologies to be economicallyjustified, while the rapid capacity increases and cost decreases have been essential enablers for the growth of the Internet.
Ultra-Long-Haul Systems
The OTrS just described requires each wavelength to be regenerated roughly every 500 km. The optical-electrical-optical(OEO) functionality required for regeneration is quite expensive; if a transponder is put on each wavelength of the OTS shown in Fig. 3.5, their cost could exceed the total costs of all the
John Strand
66
MdDemux and OAs combined. Hence extending the regeneration distance is an appealing possibility. Raman amplification, strong forward error correction (FEC), and advances in dynamic power management are making this possible. The resulting “ultra-long-haul” technology can support OTrS with dozens of 80-km spans, leading to regeneration distances of 3000 km or more. However, this comes at a cost: To get longer regeneration distances, the perwavelengthmultiplexing, demultiplexing,and amplificationscosts are likely to significantlyexceed those of traditional systems. We can expect ultra-long-haul systems to be competitive, therefore, only for long systems where these perwavelength costs can be counterbalancedby regeneration savings. A strawman ultra-long system based on 2000-2001 products is shown in Fig. 3.7. This system shown in Fig. 3.7 illustrates a number of architecturally interesting features we shall return to in the architecture discussion. a
Adaptation. The boxes labeled “A” are adaptation functions. Using an OEO transponder function, they map one or more inputs (the a’s, typically standard short-reach signals) into a long-reach OCh or group of OCh‘s that will pass transparently to a distant adaptation function. Adaptation options include: a. Multiplexing. Either electrical or optical TDM may be used to combine the inputs into a single wavelength. This is done to increase effective capacity. After multiplexing, the combined signal must be routed as a group to the distant adaptation function. b. Adaptation grouping. In this technique, groups of k (e.g., 4) inputs are managed as a group (an “adaptation grouping,” increasingly called a “wave group”) within the system and normally must be addeddropped as a group. Tight spacing is used for wavelengths within a group, with larger guard bands between groups. Grouping is done to simplify power management. It may also be possible to largely contain nonlinear effects such as four-wave mixing within the groups. Wavelength spacing may vary between groups, as may the
X
Y
PADM
PADM
6 11
Olk
D
........................ =a1
AN
T..............F IA
=Nl
=Nk
oak
Fig. 3.7 Strawman ultra-long-haul optical transport system (OAs not shown).
3. Optical Network Architecture Evolution
67
number of wavelengths in each group. Note that either option places limits on connectivity (which 0’s are delivered to which output port), because the inputs involved must be routed as a group to a compatible output adaptation function.
0
Photonic add-drop multiplexers (PADM). X and Y are “photonic add-drop multiplexers”-the all-optical analog to the SONET ADM discussed in Section 3.1. They allow economical dropping and insertion of limited numbers of adaptation groupings without requiring demultiplexing and remultiplexing all the other wavelengths. Depending on the filtering architecture, it may be possible to reuse frequencies so that, for instance, the same frequency could be used for a D to X grouping, an X to Y grouping, and a Y to E grouping. “Domain oftransparency.” The dotted line encloses an all-optical “domain of transparency,” an all-optical subnetwork. The adaptation functions just discussed optically isolate the domain.
2.2.3. More Complex Domains of Transparency Since the PADMs in Fig. 3.7 are all-optical, it is possible to build more complex all-optical domains, as shown in Fig. 3.8. In Fig. 3.8, the basic ultra-long system D-X-Y-E from Fig. 3.7 has had branches added at the PADM’s X and Y, with further branching at PADM U. In this configuration, there is an all-optical path, A-Y-X-U-Z, connecting A to Z . Transponders to optically isolate the domain would need to be present on the boundary of the domain and would serve to define the boundary of the domain. There are no “loops” in Fig. 3.8. If a U-Y link were added, the domain would turn from a topological “tree” (only one path between any two points) into a more general “mesh.” The alternate paths that result might be useful for restoration, but they also might complicate considerably the management of impairments.
OPADM
Fig. 3.8 Larger domain of transparency (OAs not shown).
John Strand
68
2.3. RECONFIGURATION CAPABILITIES
2.3.1. Why Reconfigurability Is Important Returning to the example given in Fig. 3.7, for fixed wavelengths, hk, and a fixed configuration of the PADMs, each input port is in effect hard-wired to some distant output port. The connectivity between ports is fixed. In fact, the set of transmitters and receivers that are tuned to a specific frequency can be thought of as a plane that cannot be interconnected within the domain to planes d e h e d by other frequencies. We will see later that this can be a serious problem in some situations. In addition, the lack of reconfigurability can make it difficult to effectively use the capacity of a complex domain of transparency such as that in Fig. 3.8. In this figure, for example, if a specific frequency is in use between D and E, then this frequency cannot be used for the path A-Y-X-U-Z connecting A to Z-it is blocked on the X-Y link. If it were possible to reconfigure this connectivity, the OTrS would in effect be turned into a distributed switch or cross-connect, which could be used for software controlled provisioning or for restoration after some types of failures. These types of functional capabilities are at the heart of the business rationale for deploying the optical network. There are a number of ways in which reconfigurability may be achieved: 0
0
0
0
Laser/receiver tunability. The lasers producing the LR wavelengths (the Ai in Fig. 3.8) may have a fixed frequency, may be tunable over a limited range, or be tunable over the entire range of wavelengths supported by the DWDM. Tunability speeds may also vary. Tunability may give additional connectivity options, and allow ports that could not otherwise be connected to do so. Wavelengthconversion. Internal to a domain of transparency, it might be possible to change the frequency of a connection, thereby in effect interconnecting the planes described earlier. This could be done by converting to the electrical domain and then modulating a laser (fixed frequency or tunable) with the signal. However, this is apt to be quite expensive and may degrade the signal. Conversion in the optical domain has been the subject of considerable research but products with functionality, reliability, performance, and cost adequate to make them attractive are not yet available (see [5] and [13] for surveys). Switchfabrics. A switching fabric could be placed in the PADM in Fig. 3.7 or a cross-connect could be inside a domain of transparency or be placed between two domains. This technology is discussed next. Adaptation grouping adjustments. If the boundaries between groupings are dynamically adjustable, bandwidth could be moved between groupings.
3. Optical Network Architecture Evolution
69
2.3.2. Optical Layer Cross-Connects (OLXC) An OLXC is the optical layer’s equivalent of the Digital Cross-Connect (DCS) discussed in Section 2.1. OLXCs can be placed in a number of places, as the three options in Fig. 3.9 illustrate. The transponders are the demarcation point between the short-reach optical signals, all at the same frequency, that are often used for intraoffice connectivity and the proprietary frequency registered longreach OChs used interoffice. For simplicity, connectivity to routers and other services and TDM equipment is not shown; consider all signals as coming in from the right in the figure and then looping back to the right through one of the cross-connect options. Options B and C really only make sense if they are all-optical, whereas Option A could have either an electrical or optical fabric. Option C (often called a “fiber switch”) is switching the very wide-band proprietary multiwavelength signals produced by WDM multiplexers. This has the advantage that it handles many fewer signals and so needs many fewer ports and a much smaller fabric; however, transmission impairments and technology and vendor incompatibilities impose complex and potentially very restrictive limits on the connections that it can establish; at present, only single-vendor DWDM-DWDM connections can be made, and even with a single vendor, technology differences (e.g., different frequency grids) can prevent other connections. Consequently, Option C does not seem to be getting much attention at present, at least for long-haul applications. Option B (often called a “wavelength selective cross-connect”) also deals with wavelengths that must conform to the proprietary frequency grid and transmission constraints imposed by the DWDM equipment. Consequently, it is not really an option at present, except in the interior of a domain of SR
LR
(1 frequency)
(N frequencies)
Multiplexed
I
U Transponders
Fig. 3.9 Optical Layer cross-connect (OLXC) options.
70
John Strand
transparency. Where feasible, however, it does offer the promise of substantial cost savings because it does eliminate transponders. If the domain implements adaptation groupings, this option could switch these groupings as an entity, thus reducing the number of ports and fabric cross-points required. Note also that in the absence of all-optical wavelength conversion only channels of the same color may be interconnected. Option A is optically isolated from the DWDM equipment and crossconnects wavelengths with a standard frequency. It thereforeprovidesthe most connectivity.It is insensitive to the optical architectures used by the DWDM vendors, and hence can sit between proprietary “domains of transparency.” It has two disadvantages, however: (1) It requires transponders on each port, these are expensive and also constrain the formats and bit rates that can be cross-connected; and (2) it may not scale as well as the other options because it requires a port per wavelength. Fault detection and localization can be done in Option A by the transponders, which have electrical access to the SONET/SDH overhead bytes. These functions are trickier in the all-optical environments of the other two options. In Option A, an important design choice is whether the transponders are functionally integrated into the DWDM, the OTS, or are stand-alone. Fast reaction to faults is really only possible if there is some level of integration; otherwise it is necessary for an alarm to be sent from the transponder through a time-consumingsequence of softwarelayersbefore restoration can be initiated. With the exception of all-optical network vendors, Option A has received the most attention to date.
2.3.3. Optical Add-Drop Multiplexers (OADM) The OADM’srole was mentioned in our discussion of ultra-long-haul OTrSs. The specifics of its rol+how many wavelengths need to be dropped, what constraints (if any) should be placed on which wavelengths can be dropped, what sort of rapid reconfiguration capabilities are needed-are not yet clear, and there are many technological choices to be made. Some of these choices are illustrated in Fig. 3.10.4 2.4. INTELLIGENT OPTICAL NETWORKS
Optical Transport Systems have been software intensive since their inception, but by and large this software has not been externally visible. This appears to be changing. The basic component technologies from which optical systems are built are usually availableto all systems integrators, so to differentiatetheir products they are turning to intelligent networking and management software. This software has the potential to allow network operators to reduce their operations costs and also to better customize their servicesfor their end users. Developed by Cedric Lam of ATBET Laboratories.
3. Optical Network ArchitectureEvolution
71
OADM I
completely flexible (any collection of wavelengths can be dropped)
I wavelength selective (constraints on wavelengths dropped) I
I
No banding each node can add/dro~
I multiple
only one adddrop band allowed per site
adddrop bands allowed per site
WavelengthsThatCan Be Dropped PerBand (5number of wavelengths included in a band)
I
ports not configurable
I
Ports remotely configurable (requiretunable laser)
I
md rn”*n..r.hlm !.r
(tiwedlaser)
I
I Wavelength Reusability
Fig. 3.10 OADM architecture choices.
Intelligence is appearing in several areas: 0
0
0
“Soft optics” internal to individual systems to allow things like dynamic power balancing and automatic discovery and reconfiguration of optical subsystems. “Network is the database” functionality that relieves the network operator of the expense of determining network state; instead, the network performs this function and makes the information available to queries. Automatic “point and click” provisioning of new connections using vendor provided software and Graphical User Interface (GUI).
These developments are potentially very attractive to network operators and are receiving a lot of attention. We will return to this topic later (Section 5).
2.5. OPTICAL FAULT MANAGEMENT SONET/SDH has very mature and time-tested methods for detecting faults and isolating the source of the problem, primarily based on electronic detection of bit errors using CRCs carried at each level of the frame overhead. A significant concern of network operators has been the possible existence
JohnStrand
12
of failure modes that are difficult or perhaps impossible to isolate by optical means alone. One can envision monitoring optical power or signal-to-noise ratio, but these analog measurements will not detect pulse distortion arising from nonlinearity and dispersion, for example. Maeda [121gives as an example an OLXC failure that resulted in the delivery to the proper port of a signal with the correct optical characteristics but incorrect digital content. This is a serious matter, perhaps even a show-stopper, for network operators, particularly as network growth and cost pressures continue to stretch operations resources ever thinner. A number of approaches are being tried to deal with the issue: 0
0
0
Persuasion. Green [47l points out that this issue was encountered when all-optical OAs replaced OEO regenerators that did electronic fault monitoring approximately every 40 miles. Anxious to realize the enormous economic and capacity benefits offered by OAs, operators were persuaded that pump power level and other optical parameters were adequate. Green hopes that history will repeat itself and make looking at the bits unnecessary. SignaZ splitting. A small amount of the optical signal could be diverted and examined electronically. Containment. All-optical subnetworks could be kept small and optically contained, with electronic monitoring of all connections entering/ leaving such a subnetwork. A related step would be to keep the topology of all-optical subnetworks simple-for example, require them to be topological “trees” without loops. (Fig. 3.8 is such a tree.)
2.6. FUNCTIONAL CONSOLIDATION Advances in silicon technology have created the possibility of integrating transport functions that traditionally had been in discrete boxes. For example, in SONET/SDH networks cross-connects and add-drop multiplexers have traditionally been physically separate network elements (NEs), as have DWDM terminals. However, it is now possible to implement a full ADM on a DCS line card. In the intercity network, so-called “optical layer cross-connects” have appeared: They have a large number (thousands) of OC-48/192 ports but an STS-1 fabric and the ability to do all the traditional ADM functions as well as provide an OC-48/192 cross-connect capability. In the metropolitan market, products (often called “multi-service provisioning platforms”) are appearing that carry this trend further, with optical ring functionality and the ability to groom individual DS-1 (1.5 Mb/sec) signals and even ATM switch and IP router capabilities added to the functionality mix in a single NE. The drivers for this trend are compelling: Significantcost, power, and floorspace savings result from the eliminationof the line cards and cabling necessary to interconnect network elements. In addition, maintenance costs tend to be
3. Optical Network Architecture Evolution
73
proportional to the number of NEs, and would therefore benefit from this trend. When functions are integrated in the same NE, layer boundaries get blurred. In the case of the OTN, it is likely that it will be difficult to cleanly separate the management of the OTN from the other transport networks residing in the same NE. If OTN software and operations cannot be cleanly separated from the massive and complex multilayer legacy transport network, the introduction of new OTN technology and functionality will be significantly impeded.
3. Service and Business Trends Changes and trends in the telecommunications services mix and also in the structure of the telecom industry will be critically important determinants of tomorrow's OTN architecture. In this section we will look at a few of the most important trends and their architectural implications.
3.1. SERVICE BASICS The breakdown of bytes of U.S. long-distance traffic by major service grouping is summarized in Fig. 3.1 1. The two bars on the left show the service breakdown at the end of 1997 and 1999, respectively. The bar on the right shows the breakdown of the growth in this period. The predominance of voice traffic even at the end of 1999 may seem surprising to some; much of this is due to differences in utilization, which are discussed later. The Internet segment is clearly growing more rapidly; in [55] it is estimated that average annual growth rates for voice are about 109'0, for the Internet about loo%, and for private line about 30%. Because of this, Internet traffic is expected to exceed voice traffic by early 2002. Note that even in the 1997-1999 period, network growth was driven by the Internet segment. After 1999, Internet growth is expected to become increasingly dominant. From an architectural perspective, this means
DataNetworks
EOY 1997
EOY 1999
1997-99 Growth
Fig. 3.11 Traffic on U.S. long-distance networks, 1997-1999 (from [SI).
74
John Strand
that new investmentwill need to be pointed increasingly to the needs of Internet trfic. The revenue perspective is quite different. A leading industry consulting group (RHK) estimates that voice revenue per megabit is seven times that for a T1 Internet connection; for the telecommunications industry as a whole, total voice revenues are projected to be much larger than data revenues for many years. Much of the voice revenue comes from a small number of very large corporate customers who are dependent on very sophisticated and complex software-based functionality in the legacy voice network, and therefore very hard to migrate to a new IP-based infrastructure without the same level of f~nctionality.~ This introduces conflicts into the strategic planning process, particularly for carriers with a large embedded base of voice customers, because there is a constant tension between investing in the future and satisfying the current customer base.
3.1.1. Ethernet We are focused on intercity transport, and so Local Area Networks (LANs) are out of scope. However, we should not lose sight of the fact that Ethernet is the dominant protocol in buildings and campuses, and that a large portion of data traffic is carried at least partly on Ethernet. Ethernet is a very rapidly evolving technology.6 It has benefited from enormous volumes and achieved unsurpassedfiber-porteconomies and is steadilyextendingits reach into metro and even long-haul networks, and therefore it bears careful watching as a potential service driver in the long-haul network in its own right, and even a technological competitor for some long-haul applications.
3.2. INTERNET TRAFFIC CHARACTERISTICS Since Internet traffic will be the driver for network growth and architectural change, it is important to understand the nature of this t r f i c . In this section we will look at some of its important characteristics.
3.2.1. Connection Bandwidth The size of the connections between routers will impact a number of aspects of the OTN architecture. For example, the necessity for leaving the optical IBM, for example, has thousands of call center agents averaging 95 sales calls and revenues of $63,000 a minute [107]. Such an operation depends heavily on sophisticated network call-management software to balance loads between call centers, among other functions. This softwm works in coordination with IBM’s internal computer telephony integration product, which provides a customer history screen-popwhen an incoming customer number is transferred to an mailable agent’s phone. See Chapter 11 on Ethernet in this volume.
3. Optical Network Architecture Evolution
75
layer at intermediate points to do TDM multiplexing at sub OC-48 rates will be eliminated if these connections are all at OC-48 or higher rates. The interfaces to voice-service switches normally are (1.5 Mb/s) DS-1 or channelized (45 Mb/s) DS-3. If there is a large community of interest between two such switches, there will be a large number of independent links. This is not the case for large routers, where a singlehigh-speed port normally is much more efficient than a number of lower-speed ports with the same aggregate capacity as the high-speed port. There are a number of reasons for this: (1) There is a significantgain in statisticalmultiplexingefficiencyin the one-port case; (2) the hardware design of routers has normally put a hard upper limit on the total number of interfaces that it can support. The effect of this on the service-providingfacilities7offered to today’s transport network has been dramatic. As recently as the late 1990s, the great bulk of demand growth was for DS-1s. By 2001, the bandwidth-weighted growth in most intercity networks is overwhelminglyfor unchannelized DS-3s and larger facilities; indeed, it is expected that the dominant size of service-providing facilities will become OC-48 (2.5 Gb/sec) and OC-192 as IP traffic starts to predominate.
3.2.2. Connection Length Connection-lengthdistributions have a subtle but profound effect on transport network architectures. For example, if lengths are short, the market opportunities for ultra-long DWDM systems (see Sections 2.2 and 4.2) and their enabling technologies such as Raman amplification are limited. The volume of voice traffic between two cities is determined by their “community of interest,” the propensity of people in them to want to communicate. This can be roughly modeled using a “gravity model,” which predicts that call volume between two cities is proportional to the product of their sizes (measured in people or total income, for example) divided by the square of the distance between them. This results in relatively short connections-the median length of an intercity DS3 carrying voice traffic, for example, is just a few hundred miles. Internet traffic, and particularly Web traffic, is quite different. Normally one has no idea or interest in the physical location of the server involved in a Web session. Furthermore, the determinants of server location are different, for example, Silicon Valley has an enormous concentration of servers because so many Web-based services were developed there. The net effect of this has been to lengthen average intercity Internet-related facility lengths by an order of magnitude compared to those deployed for voice services.
A “service-providingfacility” is a circuit that terminates on a service-providing NE, such as a voice switch or a router.
John Strand
76
R
Z
(a) Toll Switching Hierarchy
(b) Internet ISP Hierarchy
Fig. 3.12 Hierarchical and nonhierarchical routing.
There are a number of other more speculative trends that may affect connection length. One is based on an analogy to the growth of the long-distance voice network. Originally, “toll” voice switches were hierarchically structured into a four-layer “tree.” This was done on a geographic basis, with parent and sibling switches connected together by relatively short trunks (called “final” trunks). This is illustrated in Fig. 3.12a. Calls were passed up the tree until a switch above both caller and called party was reached. In Fig. 3.12a, an A-Z call was routed A-B-C-R-X-Y-Z. However, if there was sufficient calling volume between two lower-level switches, some “high-usage trunks” (B-Y in the figure)would be built between them to avoid the cost and delay of followingthe rigid hierarchy. These trunks tended to be much longer than the final trunks. As this network scaled, the proportion of calls handled on these trunks rose, and the higher-level switches in the hierarchy lost their importance. Eventually they were not needed, and the hierarchical structure was replaced by a flat nonhierarchical arrangement. In this process, connection (trunk) lengths increased substantially. The Internet Service Providers (ISPs) that compose the Internet are today also structured hierarchically into local, regional, and national (nondefault) ISPs (Fig. 3.12b). As the Internet rapidly grows, it would be natural if lowerlevel ISPs would discover opportunities to build the optical equivalent of high-usage trunks that would avoid the expense and delay associated with the current structure. If this occurs, one would expect the net effect to be the further lengthening of Internet facilities.8
3.2.3.
Tkaffic Symmetry
A fundamental element of current TDM-based architectures is the symmetry of the connections supported: There is always the same unidirectional
*
The drivers for “direct peering,” as this process is called, are discussed in [59], and its sigmficance is discussed in [58]. Currently a Web fetch requires two to seven round-trips to the server, each of which goes through a large number of routers (17 is the number given in [60]). By reducing the number of ISPs and routers in series, direct peering also has performance and reliability benefits that are encouraging this trend [60].
3. Optical Network Architecture Evolution
77
bandwidth dedicated to A to Z traffic as there is no traffic in the reverse direction. This originates in the design of traditional voice circuits, which are 64-kb/sec full-duplex connections. If this were to change, there would be opportunities to build more economical networks if asymmetry were better supported. There is no need for data traffic to be symmetric. A file transfer is unidirectional; on the Internet, the prevalent client-server architecture leads to large amounts of data sent from servers in response to small requests. Various studies have found twice as much traffic flowing from the United States to other countries, and much larger imbalances in flows from ISPs that specialize in supporting servers to those that are residentially oriented (see [48] and [29]). 3.2.4.
Utilization
By utilization, we mean the proportion of bandwidth that is actually used. Because there is hourly, daily, and seasonal variation in traffic intensity, it is appropriate to look at a time-averaged utilization. Odlyzko [40, 571 estimates that Internet backbones are about one-third as utilized as U.S. long-distance switched voice networks, and that private line data networks are only about 10% as utilized. There are many reasons for this that are discussed in the references; two of these provide opportunities for the OTN architecture to add value: 0
0
If it is possible to add capacity very rapidly, there is little reason for an ISP to keep a spare capacity buffer. However, if additions take a long time, it is prudent to order capacity so it is available well before it is likely to be needed to guard against unexpectedly rapid demand growth or unexpected delays in getting the capacity online. The higher the demand volatility or the more uncertain the capacity delivery process, the earlier a prudent manager will order new capacity. The Internet is noted for wild demand surges, and the time it takes to get an additional large (multi-megabidsecond) connection installed today is frequently measured in months, and there can be significant uncertainties, particularly when multiple operators are involved (e.g., a local telco and a long-distance telco). Hence early capacity ordering, which leads to low utilization on average, is standard operations procedure for many data network managers. Intranets and other business-oriented private data networks have usage concentrated during business hours. Because these networks use dedicated private lines, there is little ability to use the idle bandwidth for other purposes during off hours. Conversely, ISPs catering to residential customers are likely to see their usage higher in the evenings and on weekends.
John Strand
78
3.3. INDUSTRY STRUCTURE
The US. telecommunicationsindustry is changing in ways that are profoundly affecting OTN architecture. This section identifies a few of the changes that affect network architecture. 3.3.1. Additional Intercity Optical Networks In the mid-1990s, there were basically three national scale OTNs in the United States, each vertically integrated into one of the major intercity service providers (AT&T, MCI, and Sprint). They accounted for about three-quarters of total intercity fiber deployment. By the end of the decade, however, they accounted for less than a third, with 39 new national carriers accounting for an equal amount [61]. This same source estimates that in 1999 there was a total of 400K route miles in long-haul networks, with an average of 46 fibers per cable. This trend has a number of architectural implications: 0
0
The new OTNs can provide a facility underpinning for nonfacility based ISPs and other service providers. The resulting service competition puts pressure on vertically integrated service providers to keep the unit cost of their OTNs competitive,for example, by introducing new technologies faster than they would otherwise have done. There are many opportunities for buying, selling, or swapping fiber, leading to a situation where competing OTN operators may share fiber in the same right of way or even the same fiber cable.
3.3.2. Additional Local Networks Hundreds of so-called “Competitive Local Exchange Companies” (CLECs) provide competition for the “Baby Bells.” Particularly in areas with a high density of telecom-intensivebusinesses such as lower Manhattan, these companies provide optical connectivity to many key customerlocations. They have more incentive to introducenew architecturesthan the incumbentcarriers. The long-termeconomicviability of many CLECs is hostage to politicdregulatory developments affecting their complex relationships with the “Baby Bells” 3.3.3. Web Hosting and Carrier Hotels Facilitiesto meet the need of carriers, ISPs, and Internet content providers are changing the geographic structure of the industry: 0
“Carrier hotels” are buildings run by third parties that meet the needs of new carriers for a physical presence in some city. They provide a
3. Optical Network Architecture Evolution
e
79
telco-grade infrastructure, so all conduit, power, environmental, and security requirements are satisfied; provide an opportunity to share costs; and facilitate interconnection between CLECs, ISPs, and intercity carriers. Web hosting sites provide similar facilities for the enormous number of servers and routers deployed by Internet content providers.
These sites provide enormous concentrations of potential business and potentially make it easier for start-up OTNs to find the service volumes they need. The “dot-com” downturn in 2001 has hit a number of them very severely, however.
3.3.4. Bandwidth Trading We mentioned earlier that carriers frequently lease dark fiber from each other. These deals are each separately negotiated by the parties to specify quality of service, any penalties for contract nonfulfillment, and other details, and the physical implementation typically requires engineering and construction to establish the physical connection. This process is time consuming and expensive. This is in marked contrast with the situation in energy markets like gas, oil, and electricity where there are enormous markets to facilitate trading. These markets are based on standardizing the product’s physical characteristics and quality, specifyinga clearing and settlement process for payments, and defining penalties for nonperformance. To facilitate trading, a “benchmark” product delivered at a specified location is used as a basis for establishing a price, and conversion factors are established to establish prices for other grades and physical delivery points. Once this is done, it is possible to establish forward markets that allow buyers and sellers to do financial risk management and also allow speculation. Markets that have been through this ‘ccommodification’y process have been profoundly changed, as anyone following the deregulation of the U.S. electricity industry is aware. Many sophisticatedand very well financed players are trying to commodify the bandwidth market: A Web search for “bandwidth trading” in January 2001 got 95,800 hits. A number of intermediaries have sprung up to help buyers and sellers of bandwidth to trade with each other efficiently. One approach is to act as a “matchmaker.” Many of these companies have Web sites where owners of underutilized capacity can post their offering^.^ A smaller number of players are actually deploying equipment to facilitatethe process. A possible architecture is shown in Fig. 3.13. An offering typically identifies the two end points, the type of circuit (OC-3, STM-1, etc.), and the price and availability date. Offerings can be either “IrrevocableRight to Use” (long-term or permanent) or on some sort of leasing arrangement, e.g., on a monthly basis.
80
John Strand
Pooling Point Carrier F
DCS
CarrierK
CLEC or
Fig. 3.13 Pooling points for OC-n connections.
A “pooling point” with a DCS and/or an OLXC provides the necessary connectivity. It might be located at a carrier hotel or a Web hosting site where colocated routers require a high volume of OC-n connections. It also could be connected by CLEC or ILEC facilities to remote customer locations. Carriers might be invited to connect together pooling points in different cities. The pooling point operator could then establish a trading operation to match buyers with sellers. In Fig. 3.13, a customer wanting an OC-n from A to Z could then use one carrier from A to B and another from B to Z . The bandwidth buyers would hope to gain lower prices through vendor competition. Colocated buyers especially could then hope for more rapid provisioning of their capacity. Among the carriers, start-ups with lots of unused capacity could be expected to gain customers. New ways to use the OTNmight also arise: A large private line network that had very low utilization at night and weekends could try to sell this bandwidth at off-peak periods to someone needing bandwidth to do computer back-ups, for example. Carriers might be able to reduce their sales and marketing expenses significantly. There are a number of issues regarding this: Competition would be increasingly price driven, and there would be pressure to provide a basic standardized product. This could make it difficult to introduce technologies and products differentiated by reliability, security, or customer service. The vertical integration of network with services currently prevalent in the telecom industry would come under pressure and new business models might be needed. Many of the criteria associated with successful commodity markets are not present. A study by the Boston Consulting Group [loll identified six critical criteria: (1) Vertical deconstruction. Vertically integrated industries are poor candidates. (2) Fragmented supplier base. Suppliers with market power can refuse to participate.
3. Optical Network Architecture Evolution
81
(3) Fragmented customer base. If there are few buyers, they can be targeted by suppliers, eliminating the need for a market. (4) Price volatility. Predictable prices reduce the need to hedge risk and reduce the incentives of market makers. (5) Common unit ofexchange. Efficient trading environments require common units of exchange and settlement contracts to fuel liquidity. (6) Delivery mechanism. A physical delivery mechanism is required to efficiently and quickly move commodities between buyers and sellers.
This study concluded that in most of these areas, the bandwidth market was not yet ripe for commodification but might be in a few years. They did see immediate opportunities in a few specific areas, especially for private lines on high-volume, capacity constrained routes. In summary, bandwidth trading is not yet a significant factor for 0 7 3 s . However, it does appear that there are short-term opportunities and incentives for companies running carrier hotels and server farms, and also for some start-up carriers, to move in this direction. In the longer term, the prospects appear rosier.lo There are also significant architectural implications: Bandwidth trading would make rapid provisioning an essential network capability, would make it harder for both equipment vendors and carriers to establish proprietary product and service improvements not incorporated in the definition of the standard traded product, and would make network interworking much more important.
3.4. OPTICAL NETWORK SERVICES 3.4.1. Services Overview It should be clear from the discussion in Section 3.1, that OTN services in the future will be targeted primarily at meeting the needs of public and private data services, and particularly IP-based services. We will generically refer to the providers of these services as “ISPs.” From an architectural perspective, an OTN architect must make a choice: (1) Build a network with premium features such as very fast restoration in the hope that it will command a premium price and profit margin; (2) build a network offering only the functionality required for “commodity” bandwidth and hope to get higher volumes and efficiencies; or (3) build a network that
lo The Web is the best place to do further research on bandwidth trading. Some market participants whose Web sites might be of interest are Band-X, Interxion, AIG, and Arbinet (all facilitiesbased) and Ratexchange, Bandwidth Market, and Bandwidth.com(nonfacilitiesbased).
Johnstrand
82
can offer both commodity and premium services. We expect that there will be network operators opting for each of these options. A fruitful way to start thinking about the service/architectureinterrelationship is to identify ISP needs, identify the functionalitiesthat might possibly be provided by the network, and then match functionalitiesto needs. 3.4.2.
ISP Needs
ISP needs axe as follows: 0 0
0
0
Price. The most important need is undoubtedly low price. Availability. ISPs are struggling to keep up with exponential growth. Therefore, the availability of bandwidth is a key need-whether it can be provided at all between the desired locations, and if so, how quickly. ISP cost displacement. Displacement of internal ISP costs is also desirable. By this we mean functionality provided by the OTN that will allow the ISP to reduce their internal costs. Some potential areas for this are: a. Reduce the cost of physical interfaces on routers. b. Provide bandwidth at the speeds optimal for the router. c. Assume some of the costs of reliability and failure recovery. d. Assume the responsibility for providing exactly the capacity required when it is required. e. Allow flexible peering with other ISPs. f. Provide network management capabilities that allow the ISP to be aware of the state of their connections at all time and to reconfigure them as appropriate. Additional revenue opportunities. Provision of capabilities that enable new services. For example, a highly reliable Virtual Private Network (VPN) offering conceivably might be based on an optical layer restoration capability. Coverage of special events might be facilitated by rapid OTN provisioning of extra bandwidth.
3.4.3. Possible Functionality Areas Possible areas of functionality include the following: 0
0
Additional connection oflerings. Higher bit rates (e.g., OC-768); different formats (Ethernet, Digital Wrapper); more flexible bandwidth configurations, such as asymmetric or unidirectional connections; flexible concatenation of standard SDHBONET connectionsto provide additional bandwidth options; and inverse multiplexing. More rupidprovisioning. Software control of optical cross-connects and other reconfigurableONES.User-Network Interface (UNI) allowing
3. Optical Network Architecture Evolution
83
direct signaling by a router or customer controller requesting immediate additional connections. Larger networkfootprint. More ubiquitous connectivity can be provided by interworking between networks and by providing OTN access from more locations. Additional restoration options. A range of restoration speeds and threat coverage; assistance in dealing with service layer failures such as router or service outages.
3.4.4. Relating Needs and Possible Functionality Table 3.3 attempts to identify possible relationships between needs and functionality. The justification for many of the entries can be found in the subsections of Section 3.4.
3.4.5. A Carrier Perspective on Service Functionality The Carrier Working Group within the Optical Interworking Forum (OIF) recently produced an “Optical Services Framework and Associated Requirements” document [43] that provides a good snapshot of current services thinking in the carrier community as it relates to optical networking and software control. What follows is excerpted from this document.’*
3.4.6. Value Statement Optical networkingpermits camers to provide new types of network services not available with other technologies, enabling sophisticated transport applications of (D)WDM based networks (featuring a variety of topologies such as pointto-point, ring and mesh). These new generation networks provide means for the improved use of network resources and the support of high-bandwidth services. Dynamic bandwidth allocation, fast restoration techniques and flow-through provisioning give birth to an assortment of services. Intelligent OTNs contain distributedmanagement capabilityand subsume many provisioning and data basing functions currently performed by carrier Operations Systems (OS). This allows the rapid establishment and reconfiguration of connections, potentially reducing provisioning times from months to seconds, thus lowering operating costs and providing the means to set and guarantee SLAs12 and QoS configured on a per-connection basis to better meet customer’s specificneeds.
I I The current author was the chair of the OIF group producing this report. It is a working text and not an official OIF Technical Report, and is not binding on the OIF or its members. l2 SLA Service Level Agreement. Defines the details of the service to be provided, particularly its availability and reliability.
Table 3.3 Relations Between Needs and Potential OTN Functionality Additional Connection Offerings Price
0
0
Asymmetric connections
Rapid Provisioning
Additional Restoration Options
Larger Footprint
Reduced network operations expense
Lower internetwork coordination costs
Shortened provisioning interval
More optically reachable locations 0 Faster multinetwork provisioning
Concatenation (e.g., OC-15)
Availability
Less expensive router line cards 0 Higher utilization
0
cost displacement
0
Higher utilization
Higher utilization
Revenue opportunities
Lower delay
0
Reconfigure for special events 0 Add temporary busy hour bandwidth
0
Higher availability 0 Lower MTBF
3. Optical Network Architecture Evolution
85
The large capacity and great flexibility of such networks enables the support of several degrees of transparency to user traffic at lower cost to the end customer. The new services expectedto be enabled as aminimumare bandwidth on demand, point and click provisioning of optical connections, and optical virtual private networks. The standardized interface between the optical layer and the higher layer data service layers such as IP, ATM, SONET/SDH enables the end-to-end internetworking of the optical channels for conveying user information of varying formats. The use of standardized protocols will make the benefits of the intelligent OTNs available end-to-end, even if several networks are involved. This document also defined business models for three types of service offerings they hoped to see supported. These were: Provisioned Bandwidth Service: Enhanced leaseaprivate line services. Provisioning is done at the customer request by the network operator. . . . This is basically the “point and click” type of service currently proposed by many vendors.. . . Billing will be based on the bandwidth, restoration and diversity provided, service duration, quality of service, and other characteristics of the connection.. . . No customer visibility into the interior of the OTN is required; however, information on the health of provisioned connection and other technical aspects of the connection may in some circumstances be provided to the user network as a part of the service agreement. . .may involve multiple networks, e.g., both access networks and an intercity network. In this case provisioning may be initiated by whichever network has primary service responsibility. . .
Bandwidth-&-Demand Service: OC-n/STM-n and other facility connections are established and reconfigured in real time. Signalingbetween the user NE and the optical layer control plane initiates all necessary network activities. A real-time commitment for a future connection may also be established.A standard set of “branded” service options is available. . . . Optical Ertual Private Network The customer contracts for specific network resources (capacity between OLXCs, OLXC ports, OLXC switching resources) and is able to control these resowces to establish, disconnect, and reconfigure opticalconnection connections.In effect they would have a dedicatedoptical subnetwork under their control.. . . Billing will be based on the network resources contracted. Network connection acceptancewould involve only a check to ensure that the request is in conformance with capacities and constraints specified in the OWN service agreement.. .. Real-time information about the state of all resources contracted for would be made availableto the customer. Depending on the service agreement,this may includeinformationon both in-effectconnections and spare resources accessible to the customer.
Johnstrand
86
4. Optical Network Architectures 4.1. INTRODUCTION 4.1.1.
Unit Costs
Architecture is basically about achieving a proper balance between costs and capabilities.A common mistake is to equate “cost” with equipment cost. This is far from the truth. For a typical traditional long-distance voice service, for example, a typical cost breakdown is as follows: Access (paid to the local exchange carrier) Network-related costs Customer care, billing, miscellaneous
35% 15% 50%
The network-related costs typically divide roughly equally between the actual carrying costs associated with the equipment and the expenses associated with running the network. It is not unusual for the hardware cost of the NEs to be 10% or less of the total costs that need to be recovered. Thus it is crucial to always consider the non-hardware-cost implications of all architecture decisions. The revenue implications are also crucial. 4.2. TRANSPARENCY In an optical network, “transparency” refers to whether, or to what degree, an optical signal passes through the network optically. In today’s (2001) socalled “optical” networks, there are actuallymany OEO conversionsand a high degree of reliance on electronicprocessing. However,the vision of many optical networking researchersincludes a much larger role for all-opticalfunctionality. In this section we will explore this issue, the outcome of which will have a large role in shaping the use of optical technologyin future OTN architectures. 4.2.1.
m e s of Transparency
There are many shades of transparency. A categorizationused in the MONET project13was: Digital transparency. Transparency to intensity-modulated digital
signals of arbitrary bit rate, frame format, and protocol. Amplitude transparency. Transparency to intensity-modulated digital or
analog signals. Strict transparency. Transparency to any optical signal. l3 MONET w a s an ARPA-sponsored project established to define and demonstrate how best achieve multiwavelength optical networking of national scale. See www.bell-Iabs.com/ project/MONET or www.darpa.mil.
to
3. Optical Network Architecture Evolution
87
Even if there is optoelectronic regeneration, there are a number of levels of transparency possible [11: 0
a
0
Regeneration with retiming and reshaping (3R). This involves acquiring the clock of the regenerated signal, and thus makes it quite difficult to handle multiple frame formats. Regeneration with reshaping but without retiming (2R). This offers bit rate and format transparency, but allows jitter to accumulate, thus limiting the number of regenerations that can be done. Regeneration without retiming or reshaping (IR). This has the worst performance but can handle a wide variety of signals, both analog and digital.
We will use the word transparency to mean no optoelectronic conversions of any kind.
4.2.2. Potential Advantages of Transparency The potential advantages of transparency fall into two major categories: a
0
Format independence. A transparent network is largely indifferent to the details of the signal being transported so long as the power levels and other optical characteristics are within bounds. This has the major advantage of allowing new protocols (and also legacy protocols such as PDH) to be easily handled.I4 Without this independence, potentially each new format or bit rate requires standards changes and hardware and software to be developed and deployed. Less expense at intermediate nodes. A transparent network does not require expensive OEO functionality at intermediate nodes; electronics is bypassed by through wavelengths.
4.2.3. Limitations on Transparency [30,45,46] Unfortunately, transparency also has some serious problems: Impairments accumulate. Various forms of dispersion, nonlinearities, polarization-dependent loss, multipath interference, misalignment of lasers and WDM filters all degrade optical signals, and many of them pose increasingly serious performance limitations as bit rates increases channel spacing gets tighter, and the number of channels increases. Individually, these limitations constrain the distance a wavelength can l4 A thought-provoking example of this is quantum cryptography, an unbreakable form of cryptography that exploits the uncertainty principle of quantum theory, but requires the polarization of individual photons to be preserved. It has been demonstrated over 23 km of fiber [62].
John Strand
88
e
e
travel without regeneration and/or lead to a relentless tightening of component and fiber requirements. Furthermore they may combine to provide unexpectedly severe impairments. (See also [64].) All-optical interoperability is problematic. This problem has several dimensions. First, performance monitoring and fault location is problematic. As discussed in Section 2.5, our optical monitoring ability is limited. In a large network, particularly one involving multiple vendors or network operators, it is essential that faults be quickly detectable and the source of the problem be quickly identifiable. Second, it is unclear how to introduce new technology, such as an upgrade from 100 to 50 GHz wavelength spacing, incrementally. It appears that the technical specifications of an all-optical network must be fixed at the time it is first deployed if costly and difficult in-service upgrades of equipment are to be avoided. Wavelength interconnection is restricted. As discussed in Section 2.3, our ability to do wavelength translation in the optical domain is inadequate at present. As we shall see later in this section, this can make it difficult to use some of the capacity of the system.
4.2.4.
Opaque Optical Networks
An opaque OTN is one where each cross-connect and each OTrS is optically isolated by transponders. In its simplest form, an opaque network is formed by adding transponders at the interfaces to the OTrS shown in Fig. 3.5 (see Fig. 3.14). Figure 3.15 shows a cross-connect in an opaque network. Typically the optical signalsbetween the transponders and the OLXC would be short-reach or intermediate-reach, depending on the loss characteristics of the cross-connect, and would all be operating at the same frequency. Note that the OLXC could have either an electrical or optical fabric.
Standard SRh
:
Frequency Registered LRh
Frequency Registered LRh
Standard SRh
4--+-- +
4---;--+
Transponders
El El
Span Mux Trai
Receivers
Fig. 3.14 Optical Transport System (OTrS) with transponders.
3. Optical Network Architecture Evolution
89
0 Transponders
Fig. 3.15 Opaque node showing transponders and cross-connect.
The strengths and weaknesses of opaque and transparent OTNs are mirror images of each other. An opaque OTN is very limited in the formats and bit rates of the signals it can carry, and it incurs significant costs for OEO functionality for each wavelength at each node. On the other hand, in an opaque network, impairments do not accumulate, interoperability is guaranteed, and wavelength translation is obtained as a by-product. All the large OTNs known to the author have found the case for opacity compelling to date. 4.2.5.
Domains of Transparency
The choice between opacity and transparency is not really black and white. The concept of a “domain of transparency”-a transparent subnetwork, optically isolated from the rest of the network by transponders-provides a means to control the drawbacks of transparency discussed above. This technique offers us the possibility of limiting the size of each domain, thereby keeping impairment-related problems in check. New technologies might be put in separate domains, thereby avoiding technology interworking problems. Organizational boundaries, such as those between network operators, can be aligned with domain boundaries. A DWDM system such as that shown in Fig. 3.14 is an example of a domain of transparency. If transponders are put on all the q, in the ultra-long OTrS shown in Figs. 3.7 or 3.8, a more interesting domain would be defined. In effect, this is what vendors of such systems are proposing. The technological trends enabling longer wavelengths and all-optical reconfigurability should make ever larger and more complex domains of transparency feasible. However, there are costs associated with introducing multiple domains of transparency. On each boundary, costly transponders must be installed. In addition, as we shall see later when we discuss control planes, additional complexity can be added to the processes that route and manage wavelengths.
90
John Strand
4.2.6. Economics of Transparency The economic attractiveness of a domain of transparency arises largely from the opportunity to reduce transponder (OEO) costs. These per-wavelength costs can be the largest single cost in an OTN, as is shown in Fig. 3.16. The cost breakdown is for an OTrS such as that in Fig. 3.14 that is bounded by transponders. The costs are based on typical vendor prices in 2000; they could vary considerably based on the vendor’s pricing strategy and the specific size and capabilities of the OTrS. The transponder costs are linear in the number of working wavelengths (utilization), but independent of the number of spans, whereas the other costs are independent of the utilization and linear in the number of spans (except for the MudDemux), hence the relationships shown in Fig. 3.16. In an opaque OTN, transponders need to be placed on the ports of each OTrS. OTrS lengths are limited by (1) technology (the maximum number of spans before regeneration is needed), and also (2) by the need for wavelengths to be added or dropped from the OTrS. To get some insight into the economic trade-offs involved in transparency, we will give an example related to (1). Consider the example given in Fig. 3.17. Figure 3.17a shows a sequence of standard OTrS and an ultra-long-haul OTrS. Both are assumed to have the same OA spacing and the same wavelength capacity. The standard OTrS we assume to be limited to a five-span configuration before transponders are required for regeneration; in (a) this happens at offices B and C. The ULH system merely needs an OA at these locations. To go further, the ULH has presumably been designed using additional costly technology (see Section 2.2). We model this parametrically by use of
n
Transponder
Optical Amplifier
I
h Utilization (%) # Spans (80 km)
50
100
3
3
I
50 7
100 7
Fig. 3.16 Cost breakdown, OTrS bounded by transponders.
3. Optical Network Architecture Evolution
OTS Type
100
Transponder
0
b
91
ULHlStandard Cost Ratio
Y
C
._
Standard
0 ULH
1 3 5 7 9 No. Of Standard OTS Systems (5 span) In Series
(a) Reference Systems
(b) Domains Of Application
Fig. 3.17 Ultra-long-haul economics.
a cost ratio (assumed the same for DWDMs and for OAs). The additional ULH costs are system costs, which are independent of the number of wavelengths that are actually equipped. The penalty for opaqueness incurred by the five-span configuration is partially per-system (the additional back-toback DWDMs required) but primarily per-wavelength (the transponders at intermediate nodes like B and C in Fig. 3.17). Therefore, the ULH system would be expected to become more competitive as the number of systems increases (more DWDMs for the competing solution) and also as the utilization increases (more transponders). This is quantified in Fig. 3.17b, which shows for various cost ratios15 the frontier between the regions where each alternative has an economic advantage. The ULH system is economically preferred above and to the right of the appropriate curve. For example, if the ULH system is 75% more expensive, the arrows indicate that ULH is preferred for fills above about 45% if 5 systems (25 spans) are needed. If the topology of the subnetwork is more complex, as in Fig. 3.8, an alloptical solution has the additional advantage that transponders are not needed at the branch points. Evaluation of this benefit is more complex and will not be discussed further here. Unfortunately, we are not aware of any efforts to systematically look at this effect. In metro areas, the economic issues are quite different. Normally OEO is not needed for transmission reasons, because the distances involved are relatively short. Instead, the ability to carry a wide variety of bit rates and formats becomes more important. 4.2.7.
Incorporating Domains of Transparency in a Network
Domains of transparency will be limited in size by transmission constraints, the inability until standards evolve significantly to have optical interworking l5
Representative year 2000 list costs for the "standard system" were used in this exercise.
92
John Strand
Fig. 3.18 Express domain of transparency example.
between vendors, and by economics. In a long-haul network, a likely initial use of a domain of transparency would be to provide an express backbone on which longer connections would be routed. Shorter connections would be routed on an opaque technology that is more economic over shorter distances. This situation is illustrated in Fig. 3.18. In this mesh network, two technologies are used, one all-optical (shown by dashed lines with squares showing offices with DWDMs or OADMs), the other one an opaque technology with electrical fabric OLXCs (circles) and with solid lines showing connectivity. On the left, the two technologies are shown with the topologies separate, and on the right, they are laid onto the conduit network used for both. As just discussed, the all-optical technology is most cost-effective for longer distances. In this case, if we wished to route a connection from A to Z, the best route might be to go from A to J using the opaque technology (top plane in Fig. 3.18a), then go J-N-P-Q-L using the all-optical technology, and complete the route from L to Z using the opaque technology. This example suggests that introducing such a domain of transparency will raise new issues for routing and probably for other operational areas. We consider routing next. 4.2.8.
Routing and Wavelength Assignment in a Domain of Transparency
Within an all-optical domain, “wavelength conversion” (changing the wavelength of a connection) is still expensive and not yet practical without an OEO conversion. Therefore, it is important to understand the architectural implications of limited (or no) wavelength conversion. This requires us to look at what is called the “Routing and Wavelength Assignment (RWA) Problem” [l]: Given one or more connections that need to be established in an all-optical domain, determine the routes over which each connection should be routed and also assign each connection a color. If the routes are already known, the problem is called the “Wavelength Assignment (WA) Problem.” The RWA problem has received extensive attention in the literature, mostly from a mathematical perspective. This literature is best approached through
3. Optical Network Architecture Evolution
93
a survey article (e.g., [l, 2, 14, 741). The underlying mathematical problem is very hard in general. The WA problem is easily seen to be equivalent to the problem of coloring the nodes of a graph so that no two nodes connected by an arc of the graph have the same color: Simply represent each connection by a node, and connect every pair of nodes whose corresponding connections ride on the same link (and so need to be assigned different wavelengths). This coloring problem is known to be NP-complete [75], which means that, in general, it is computationally intractable, but there are many fast but approximate (heuristic) algorithms for solving it. Before discussing some of the architectural implications of this problem, it should be emphasized that the results in the literature frequently depend crucially on subtle points of the problem definition. The followingdefinitional choices are particularly important: 0
0
0
Is there an accurate forecast of future connection requirements? If so the RWA process can take a more global approach to decision making. What is the expected holding time of a connection? Some analyses assume that connections are permanent, while others assume that holding times are quite short. Is it possible to rearrange connections?In situations where unexpected demands are likely, this makes a major difference.
We feel that the most appropriate assumption at the present time is to assume that (1) we have minimal knowledge about future demands, therefore each demand must be routed and assigned a wavelength when it appears without knowledge of future demands, and (2) that holding times are very long. With these assumptions, we simulated a number of RWA algorithms on a realistic model of the U.S. intercity network [76] without rearrangement or wavelength conversion and reached the following conclusions: 0
0
0
It is important to choose the route and do the wavelength assignment simultaneously.If routing is done first and then a wavelength is assigned, significantly suboptimal results are obtained (e.g., low utilization). Even with simultaneous route and wavelength selection, a significant capacity penalty can be incurred if wavelength conversion is not available. This penalty can be significantly mitigated if wavelength conversion is available at a small subset of nodes. In particular, half of the lack-of-wavelength-conversionpenalty is removed if 21% of the nodes have conversion capabilities; 75% was removed if 35% of the nodes have conversion capabilities.
John Strand
94 e
Dual-OTrS scenarios, which provide two instances of each wavelength on each link, reduced this penalty by only 6%.
A recent survey of the value of wavelength conversion [74] concluded that the performance improvementsit offered depended on a number of factors, including the network topology and size. It further concluded that in networks with tunable transmitters and receivers, limited wavelength conversion provides improvements close to that achieved with ideal wavelength conversion. 4.2.9.
Effect of Impairments on Routing in a Domain of Transparency [19,95]
As domains of transparency get larger, optical impairments such as amplifier spontaneous emission (ASE) and various types of dispersion may become an issue. Specifically: (PMD). PMD imposes a limit on the maximum wavelength length that is inversely proportional to the square of the bit rate of the signal. For typical installed fibers, the limits are 400 km and 25 km for bit rates of 10 Gb/s and 40 Gb/s, respectively. With newer fibers assuming PMD of 0.1 ps/,/km, the limits are 10,000km and 625 km, respectively. e AmpIiJierSpontaneous Emission (ASE). ASE imposes a constraint on the number of spans that is inversely proportional to its optical bandwidth. e other polarization-dependent impairments. For example, many components have polarization-dependentloss (PDL) [11 that accumulates in a system with many components on the transmission path. The state of polarization fluctuates with time, and it is generally required to maintain the total PDL on the path to be within some acceptable limit. a Nonlinear impairments. As wavelengths get longer, nonlinear impairments such as four-wave mixing cause more problems at a given launch power.
e Polarization Mode Dispersion
These constraints are summarized in Fig. 3.19. The specific constraints required in a given situation will depend on the design and engineering of the domain of transparency. For example: e
e
The effect of nonlinear impairments depends on complex factors, e.g., on the order in which specific fiber types are traversed, the specific types of fiber involved, and the characteristics of the other active wavelengths (see, e.g., [46] or [64]). The impact of chromatic dispersion may depend on whether it has been dealt with on a per-link basis, and whether the domain is operating in a linear or nonlinear regime.
3. Optical Network Architecture Evolution \
95
PMD Constraint
Launch Power (PL)
Bit Rate PMD Parameter Optical Bandwidth Minimum acceptable SNR at receiver
**
L6ther System Parameters
Length Of All-Optical Path
Fig. 3.19 Effect of impairments on wavelength muting.
4.2.10. Connectivity Limitations Associated with a Domain of Transparency The strawman ultra-long-haul system shown in Fig. 3.7 may be thought of as a large distributed switch. Depending on the configuration of the OADMs and the tunable lasersh-eceivers, the port-to-port connectivity of the a's will change. However, the connectivity is limited: 0
0
0
The adaptation function forces groups of input channels to be delivered together to the same distant adaptation function. Only adaptation functions whose laserdreceivers are tunable to compatible frequencies can be connected. The switching capability of the OADMs may also be constrained. For example: a. There may be some wavelengths that cannot be dropped at all. b. There may be a fixed relationship between the wavelength dropped and the physical port on the OADM to which it is dropped. c. OADM physical design may put an upper bound on the number of adaptation groupings dropped at any single OADM.
For a fixed configuration of the OADMs and adaptation functions, connectivity will be fixed: Each input port will essentially be hard-wired to some specific distant port. However, this connectivity can be changed by changing the configurations of the OADMs and adaptation functions. For example, an additional adaptation grouping might be dropped at an OADM or a tunable laser retuned. In each case, the port-to-port connectivityis changed. This capability can be expected to be under software control. Today the control would rest in the vendor-supplied Element Management System (EMS), which in turn would be controlled by the operator's 0%.
John Strand
96
4.3. LAYERING AND THE OPTICAL LAYER The term “optical layer” is frequently used in architecture discussions. The term “layer” is taken from a modeling methodology that must be understood if one wants to read the technical literature on this subject or work on network architecture problems. This section gives a very brief overview of the methodology and then applies it to optical networks. The methodology described in this section was developed by the International Telecommunication Union (ITU), the international governmentsanctioned international standards organization. Reference [65] defines the basic approach, and [66] and [67] apply the approach to SDH and OTNs, respectively.
4.3.1. Layering Concept A transport network may be vertically decomposed into a number of layers that are related by a client-server relationship as shown in Fig. 3.20. Only two layers are shown. The upper layer is the client: It requests services from the lower layer, where a “service” is defined by its protocol, bandwidth, service quality, and perhaps other functionality. The lower server layer, in turn, provides capacity and the other aspects of the desired service. A couple of caveats about layering: 0
0
There is no single way to define layers. For example, layered views of the Internet normally collapse all the transport layers into one, whereas transport-oriented layerings normally collapse the multiple protocol levels describing the various Internet protocols into a single layer but show many transport layers. There is no necessary relationship between layers and hardware. A single box may perform the functions of several layers or a single layer may be implemented in a set of boxes.
Well-Defined Interface Protocol Service Quality Functionality
-
Requested
Capacity Layer N
Fig. 3.20 Layering concept.
3. Optical Network Architecture Evolution
97
4.3.2. Transport Layering The layers in today’s transport network are shown in Fig. 3.21 and Table 3.4. The relationship between these layers is illustrated in Fig. 3.21.
4.3.3.
Layers and Planes
To function, a transport network needs a signaling and control infrastructure. These infrastructures are also layered, as illustrated in Fig. 3.22. The shaded stack labeled “User Plane” corresponds to the layers we have been discussing. Two additional “planes” are shown in the figure: Control plane. Responsible for activities such as connection set-up or tear down and restoration. Operates in a dynamic “real-time” environment that directly responds to user activities or network state changes such as failures or maintenance activities. Increasingly its functionality is distributed and implemented in software resident on each ONE or directly connected to it. It has its own infrastructure for signaling to other systems within the network or to users or other networks. It is important to note that there may be separate control planes for each layer, as shown in Fig. 3.22. Management plane. Gives the network operator visibility into and control of the other two planes. Normally implemented in centralized OSs with information exchange carried over operations communications channels. This results in a static control environment characterized by scheduled activities and delayed response to network state changes.
DSl(1SMbls) DS3 (45 Mbk) STS-3 (155 Mbk) STS-12 (622 Mb/S)
igital Transmissio
STS-48~(2.5 Gb/S)PTP 1 n?- fin P . I . L \ *,*-,7LL (‘VU”,>,
Proprietary (20 Gbk -400+ GbA)
Fig. 3.21 Transport layering.
Table 3.4 Transport Layers Layer Services
Sublayer
Digital Transmission
Typical Demand
Typical Nodes
Typical Links
Voice services
calls
DSO
Circuit switch (4E)
Data services
Data transfer
Packets
Packet, frame, cell switches
Private line DSO Private line DS 1
DSOs DSls
DCS-1/0 W-DCS
DS 1 DS3, STS-I, STS-3
Private line DS3/STS3/STS12 Switch access lines, private lines Switch access lines, private lines
DS3s
B-DCS
DS3, STS-N
Digital Cross-Connect DCS 110
WS)
Service Requests Originatingat Layer
Wideband DCS (W-DCS) Broadband DCS (B-DCS) Self-healing rings Linear add/drop or point-to point systems
Optical
See text
Private line STS-48/STS-192
Media
Fiber
Dark fiber
DS3, STS-N Add-drop multiplexer
DSO “trunks” (Modulo DS1) DS0/1/3, STS-N(c)
DS3, STS-N Terminal multiplexer, add-drop multiplexer
OC-12,OC-48, OC- 192 OC-3,OG12,OC-48, OC-192
STS-48c, Optical ADM, optical STS-1 9 2 ~ cross-connect DWDM OTS
Optical transport systems (OTrS) Fiber pairs
3. Optical Network Architecture Evolution
99
N Layers 3
.: * ‘ <
5
< <
2
1 Physical Medium (fiber)
Fig. 3.22 Abstract layered network showing control and management planes.
4.3.4. Sublayers of the Optical Layer The sublayers of SONET and SDH were described earlier (Section 3.1). The ITU is in the process of defining a very similar structure for what they call the “Optical Transport Network” (OTN). Starting from the top, the ITU’s layers are: e Optical Channel (OCh) Layer. Provides end-to-end networking of
optical channels for transparently conveying information of varying format such as SONET or G-Ethernet. In non-ITU circles, an OCh may be called by other names, of which “lightpath” seems to be the most popular. Today OCh’s typically carry STS-48 or STS-192 signals, with Gigabit- or 10-Gigabit Ethernet also receiving considerable attention along with STS-768. OLXCs provide cross-connect functionality for this sublayer. e Optical Multiplex Section (OMS) Layer. Multiplexes OChs into the multiwavelength optical signal that is passed between DWDMs. It also is responsible for wavelength shifting and management. A fiber switch (see Section 2.3) might provide cross-connect functionality for this sublayer. e Optical TransmissionSection (OTS) Layer. Provides basic transport functions for an OMS such as amplification and gain equalization. Overlaying this structure on Fig. 3.1 one gets the system shown in Fig. 3.23. In SONET and SDH, at the terminations of the equivalents of each OCH, OMS, and OTS, the signal is electricallyaccessible and so it is straightforward to add overhead bytes for management and signaling. This is not the case for the OMS and OTS, therefore the ITU is proposing an out-of-band optical “OTM Overhead Signal” (00s).In [69] the logical elements to be included are specified but the format is not. Since today OTSs are invariably singlevendor, the handling of OTS and OMS overhead is vendor proprietary. There is, however, some interest from carriers, particularly in Europe, for all-optical
100
John Strand
interworking between vendors. This may ultimately make standardizing of the 00s of more interest.
4.3.5. Optical Channel Format Issues Optical channels (OCh) are bounded by 3R regeneration, and therefore are electrically accessible. This makes in-band overheads possible. ITU’s draft G.709 standard [69] has defined a frame structure to do this based on earlier proposals for a “digital wrapper” [3]. It encapsulates the payload (e.g., a SONET frame) within a larger frame containing fields reserved for framing, inter-ONE communications, and mechanisms for performance monitoring and propagating error information both forward and backward along the channel. Protection switching has been left for further study. A specific forward-error-correction(FEC) algorithm, a Reed-Solomon RS(255,239) code, is also specified. The proposed standard has a number of attractive features: 0 0
0
0
It is explicitly targeted at optical transport networking. It starts to provide the basis on which wavelengths or OChs can be effectively managed. This makes more realistic the elimination of service-specificequipment and processing inside the OTN, and thus brings all-optical networking closer to practicality. It opens the door to eventually phasing out SONETISDH functionality and eliminating a layer from the protocol stack. Unlike SONETISDH, it is not optimized for traditional voice services and arguably is less inhibiting for packet-over-fiber applications such as Gigabit Ethernet and 10-Gigabit Ethernet over fiber.
However, whether this standard will eventually be widely adopted is unclear at this point: 0
A major driver for the approach taken is the assumption that there will be a wide variety of client data networks with varying protocols and
3. Optical Network Architecture Evolution
0
0
0
101
service needs [3]. The recommended approach is designed to allow a wide variety of client protocols (SONETEDH, ATM, IP, PDH, etc.) to be transported independent of their format. There are, however, many who argue that IP will become the common traffic convergence layer for all services. If this does indeed happen, the importance of flexibly handling a wide variety of protocols will decrease and a less general, IP-centric solution may emerge. FEC needs have not yet stabilized. For example, ultra-long-haul vendors are using strong FEC algorithms that require more FEC information than allowed in the standard. The timeliness of trying to standardize FEC now is therefore an issue.I6 At present, optical interworking between vendors is limited to intraoffice short-reach connections covered by SDH/SONET standards. In long-haul networks, vendors would be optically isolated from each other by transponders. In this architecture, FEC originates and is terminated by a single vendor, and therefore proprietary solutions are feasible and allow vendors to seek competitive advantage with innovative FEC solutions. SONET/SDH can be used to encapsulate a wide range of protocols including IP, ATM, and Ethernet, thereby preserving the large investment in SONET/SDH transponders and 0%.
This is not to say that the proposed G.709 standards will not be widely adopted eventually; however, their future seems most promising if multiple data protocols flourish, FEC technology stabilizes, practical multivendor all-optical standards emerge, and rapid growth (which would allow network operators to justify writing off their investment in SONET/SDH transponders and line cards) continues. In the meantime, a vendor wanting some of the promised functionality might wish to use the proposed format within a single domain of transparency. This would avoid most of the objections listed previously, because at the boundary of the domain, an encapsulation of SONET/SDH into the new format could be done without necessarily affecting the larger network or service level interfaces. 4.4. EVOLVING ROLE OF THE OPTICAL LAYER 4.4.1.
SONET/SDH and WDM
One of the most important functions performed by SONET/SDH has traditionally been TDM-based intermediate multiplexing. This function is possible l6 G.709 is sensitive to this issue. There is an appendix that discusses alternatives to the recommended approach.
John Strand
102
only if the SONET line rate is significantly larger than the data rate of the underlying service. For traditional 64-kb/sec voice services riding on 1.5-Mb/sec DS-ls, this is certainly t r u e - a STS-48/STM-16 can carry 1344 DS-1s. However, this is not true for large IP backbone networks, where routers are increasingly connected with STS-48 or STS-192connections. The situation is illustrated in Fig. 3.24, which shows the maximum availableTDM rate (solid line), single-fiber WDM rate (dotted line), and IP router fabric capacities. The gap between the line speed of IP routers and the maximum TDM-line rate represents the opportunity for TDM intermediate multiplexing. It has basically vanished and the prospects for it catching up again are uncertain at best. Because data trailicis driving network architectureevolution, this implies that SONET/SDH intermediate multiplexing capabilities will be of declining importance. Because of this change, the SONET networking layers (the cross-connect and digital transmission layers in Table 3.4) cannot perform their traditional role for much of the bandwidth growth expected to be generated by IP-based services. Instead, their functions need to be distributed between the IP services layer and the optical layer, as illustrated in Fig. 3.25. This redistribution will very likely have major impacts on equipment vendors as well as network operators. If the functionality largely goes into the IP layer, then vendors of IP equipment will be well positioned to control transport evolution and also to charge premium prices; if it largely goes into the optical layer, then the OTN vendors will be in the driver’s seat. The rest of this chapter largely is devoted to spinning out the implications of this change. There are three major functionality groupings that must be studied:
100 L
8
lo
1
..TDM Multiplexing 0pp-v
0.1
Fig. 3.24 The declining opportunity for TDM multiplexing.
3. Optical Network Architecture Evolution
103
DS 1 (1 5 Mb/s)
STS-12(622 Mbis)
F D i g i t a lTransmissiox
I I
-.
Restoration
A
Netwnrkinn
x
STS-192c (10 Gb/s)
I rl ‘
Propnetary (2OGb/s--400+ G
Fig. 3.25 Redistribution of SONET functionality for IP services.
0
0 0
Physical Layer Functions. These include framing and fault detection and isolation. Restoration. Recovering from a failure. Networking. Provisioning and network management.
4.4.2. Future of SONET Physical Layer Functions Even if the traditional SONET networking functions are no longer needed, it does not follow that SONET technology will not continue to be used for framing, fault detection, and localization, and all the other operations functions currently based on SONET. IP routers produce SONET-formatted signals, for example, and there is no reason why they could not continue to do so. The key arguments against this are: 0
0
0
SONET overhead imposes a “bandwidth tax” of approximately 4%, which is no longer justifiable. SONET chip sets are much more expensive than competitors for other technologies like Ethernet. The elaborate operational processes built over the years around SONET are no longer needed if its networking function disappears.
The force of these arguments will vary by network and application. In metro area networks, for example, Ethernet is already a strong competitor for
104
John Strand
SONET for the transport of data between LANs. At the other extreme, an established carrier with a large investment in SONET equipment and SONET-based operational processes is unlikely to be tempted to migrate legacy TDM-based services to a non-SONET infrastructure. The “bandwidth tax” argument is not very compelling. The 4% overhead is significantly less than the overhead associated with higher-level protocols, for example, TCP/IP alone has a 15% overhead on an average-sized packet.17 A number of alternatives to SONET framing have been proposed: 0
0
0
There have been a number of proposals to create a “SONET Lite” by stripping out all SONET networking functionality. However, several vendors reported to the Optical Interworking Forum that doing so would only save a few dollars per OC-48 chip set. Ethernet components benefit from enormous economies of scale because of their ubiquitous use in LANs. As mentioned earlier, this makes them a formidable competitor, particularly for LAN interconnects over relatively short distances. At present, Ethernet does not have the operational robustness that established intercity network operators require; however, it would be surprising if one of the start-up carriers did not try to differentiate themselves with a low-cost Ethernet-based network. In the optical layer, the overhead proposed for the OCh (discussed earlier) could be a solid foundation for a SONET replacement, particularly within a domain of transparency (as discussed earlier).
Discussions of the alternative protocol stacks that could be used for IP in an OTN are discussed in [3] and [4]. The protocol stack alternatives are summarized in Fig. 3.26. In this figure, “Optical Layer” refers to OLXC-resident functionality for managing OCh provisioning and for restoration.18Additionally, it may include the OCh framing functionality discussed previously. (It is discussed in more detail later in this chapter.) “ATM” refers to Asynchronous Transfer Mode, a cell (fixed-sized packet) technology used frequently today in large IP networks to allow better traffic engineering and network management than is currently available in IP. Multi-Protocol Label Switching (MPLS) provides ATM-like capabilities within the IP framework. Each of the layers normally has its own management layer. In Fig. 3.26 “IP/MPLS” indicates that either IP alone or IP with MPLS might be used.
l7 Mean packet size measured in [48], for example, is roughly 300 bytes including headers; IPv4 and TCP per-packet overheads are at least 20 bytes each. [4] puts the OXC in the “Physical Layer” and limits the “Optical Layer” here to what we will describe later as the “optical control plane.”The definitionused here is more compatible with the terminology used elsewhere in this chapter.
’*
3. Optical Network Architecture Evolution
IP
0
0
0
0
105
(b)
ATM
IPMPLS
SONET
SONET
IPMPLS
Optical Layer
Optical Layer
Optical Layer
(C) (d)
IPMPLS
Stuck (a) is common today. It has the most framing overhead and the most management layers. Stuck (a) is often called “Packet over SONET.” It eliminates one level and the associated overheads and management systems. It also is commonly used today. (See [3,4] and [70].) Stuck (c) is what is often called “IP over WDM.” It further slims down the protocol stack and associated management systems. To be successful in large networks, the fault management and maintenance capabilities of one or both of the remaining layers need to be beefed up. Stack (d) is the simplest. It might be implemented by integrating a WDM terminal directly into an IP router. All the residual SONET/SDH fault management and operations capabilities will need to be assumed by IP/MPLS.
4.4.3. Architectural Role of Optical Layer Switching An office configured with and without an optical layer switch is shown in Fig. 3.27. Note that from an equipment cost perspective, the hard-wired office will be less expensive because it has one less network element. However, in a hard-wired office, each time a wavelength is added to the office, manual cabling must be installed between the two appropriate OTrS (through wavelength) or between an OTrS and the appropriate router or other service layer equipment (terminating wavelength). This is expensive, error-prone, and slow.l 9 It also is difficult to do this before a specific service request defining which ONESneed to be connected is received. If there is a switch in the office, as long as care is l 9 It is especially problematic for the increasing number of “dark offices”-locations that are normally unstaffed. For these offices a special visit must be scheduled if hard-wiring is required.
106
John Strand
n Service Layers
(a) Hard-Wired Office
(b) Office With Switch
Fig. 3.27 Office configurations with and without a switch.
IP Router
(a) OLXC Office Architecture
(b) “BFR Office Architecture
Fig. 3.28 Optical Layer switching alternatives-OLXC and “big fat router.”
taken to ensure that a spare connection is always available from each ONE to the switch, none of these difficulties arise-the software-controlled switch can establish the desired connectivity on demand. Arguments along these lines seem to have led most carriers to conclude that the option in Fig. 3.27b is desirable in larger offices at least. In Section 2 we introduced optical layer cross-connects (OLXCs). These are certainly a candidate for use here. The principal competition for an OLXC-based architecture is likely to come from “big fat routers” (BFR; [80, 8 11). In this architecture, IP routers with line-side interfaces operating at the per-wavelength bit rate are directly connected to one another via pointto-point OTrS. The OTrS terminals might even be integrated into the IP router. Legacy services requiring constant bit-rate connections are supported with virtual leased-line services [84]. The two competing architectures are shown in Fig. 3.28. The BFR approach is most appealing if IP-based traffic continues its dramatic growth and dwarfs other types of traffic. In this scenario, it is argued that there is no need to interpose another layer between the OTrS and the IP routers. It is further argued that eliminating a layer of switching will simplify network management and will reduce capital costs by eliminating some “boxes.”
3. Optical Network Architecture Evolution
107
Ports 8,Assumed Costs OLXC $x d IP Router $y
E4
Terminating Through (a) OLXC Office Architecture
(b) "BFR Office Architecture
Fig. 3.29 OLXC and BFR configurations.
The counterarguments in favor of the OLXC-based architecture are, briefly, (1) non-IP-based services are expected to be very important to carriers for some time (see Section 3.1); (2) it is not clear whether router technology will be able to scale to port counts consistent with multi-Tb/s capacities without unduly compromisingperformance, reliability,restoration speed, and software stability [80, 821; and (3) the OLXC architecture will have lower capital costs than the BFR architecture, even for IP traffic. We will return to the management issues when we discuss the control plane (Section 5). Here we look only at the capital cost comparison. The prices of fully loaded large routers and OLXCs are both dominated by line-card costs. We will therefore ignore other costs. For simplicity we will also assume all traffic is IP. Figure 3.29 identifies the cost elements in each architecture for terminating and for through connections. If each OLXC port is priced at $x and each IP router port at $y, and the proportion of connections that are through is a, then the average costs per connection are:
OLXC:a(2x)+ (1 - a)(2x +y )
+
BFR: 4 2 ~ ) (1 - a)b)
Solving, the OLXC architecture is cheaper if x < ay.'O A typical value of a in a long-haul network would be 0.8, so OLXC line cards must be 20% cheaper than IP line cards to be competitive. Is this likely? Based on representative year 2000 prices from a number of vendors, the answer is definitely yes. The x / y ratio for electrical-fabricOLXCs was comfortably less than 0.5; [ll] quotes a value (1999) of 0.22. The ratio is expected to become more favorable to OLXCs when transparent OLXCs appear. A technology comparison makes this relationship quite intuitive: In a transparent OLXC there is virtually no line card functionality needed, while IP router architects have achieved scalability by in effect making each line card *O A more comprehensive theoretical analysis looking at various network topologies may be found in [83]. The conclusions reached are consistent with ours.
108
John Strand
a mini-router. We conclude that an OLXC is the most economic way of doing optical layer (OC-48/OC-192) cross-connections. 4.5. SUR WABILITI: PROTECTION,AND RESTORATION 4.5.1.
Background
Survivability-the ability to deal with and recover from failures-is a critical architectural consideration. In Fig. 3.25 each of the layers shown, except the Media Layer, have survivability capabilities Unless carefully designed, this functional overlap introduces additional costs and coordination issues. The protection and restoration functions provided by SONET/SDH are critical to the survivabilityof today’s TDM-based transport networks. Before discussing how these functions relate to the IP and the Optical Layers, we will take a digression to introduce the general topic of survivability. There are many types of failures that can occur in an optical network, but two are of particular importance: equipment failures and route failures. Equipment failures, primarily failures of individual components, such as a laser, are far more numerous than route failures. In “carrier class” equipment, normally there are no single points of failure; if a component such as a laser fails, automatic protection switching (APS,discussed later) is used to restore service. Although much less frequent, route failures (also called fiber cuts) can have a much more devastating effect. The prototypical cause of a route failure is an errant backhoe that physically severs the fibers; other causes might be a natural disaster (flood, earthquake) or a train wreck. Normally all fibers in a cable will be affected. By electronic standards, these failures occur in slow motion. Initially fiber stretching might cause a period of transient errors; the actual severing of fibers can be spread out over a period of seconds. This can cause confusion and churning if a recovery process able to react in milliseconds locks in on a restoration plan before the full magnitude of the failure is known. The incidence of route failures varies widely. Rural areas have fewer problems, as do cables run through concrete conduits. As a rough rule of thumb, one fiber cut per thousand route kilometers per year might be used; thus a national-scale network in the United Statesmight expect, on average, a couple of incidents per month.21 Physical repair after a route failure requires dispatching a repair crew to the site, which may be quite distant, and then physically splicing the ruptured In 1997, when the total intercity fiber in the United States was 59,000 route miles, 136 fiber cuts were reported by various US.carriers to the Federal CommunicationsCommission [6, 711. Only cuts with serious service impacts are required to be reported, therefore the total number of cuts is no doubt somewhat higher.
3. Optical Network Architecture Evolution
109
fibers. The nominal time for doing this is on the order of 4 hours, but it may be much longer if there is a massive failure, for example, a building fire or an extensive wash-out. For many services, however, the overall availability requirements are on the order of 99.999% (6 minutedyear) or better. Therefore, additional survivability mechanisms must be provided. There are many possible mechanisms, some of which are appropriate for service restoration and some of which need to be resident in a transport layer. Because of their importance in transport networks, we will focus on recovery from route failures in this section. Optical Layer mechanisms will be discussed first, then mechanisms appropriate for other layers will be introduced and related to the optical layer. 4.5.2.
Protection and Restoration
Protection and restoration are both used in the process of recovering from a failure. A quick review of a few published articles found a number of dehitions, of which the most popular were as follows: 0
0
0
0
Protection refers to the primary recovery, whereas restoration refers to secondary methods that are invoked if the primary method fails. Protection refers to methods whose operation is specified prior to a failure; restoration refers to methods that dynamically determine how to proceed. Protection reroutes onto preassigned spares; restoration makes use of a pool of spare resources that are dynamically assigned after a failure. Protection refers to recovery from a component or subsystem failure by switching to a hot standby that is usually performed in a distributed way by the ONES.Recovery that requires rerouting facilities over a physically different route is called “restoration.” Protection is triggered by “defects”-the physical layer’s detection of a problem, whereas restoration is triggered by ‘Lalarmsyy-messages generated by ONESto be sent to external managementlcontrol functions.
In general, “protectionyyhas historically referred to deterministic methods that are very fast, whereas “restoration” referred to more complex methods whose precise behavior is determined at the time of failure, and which may be somewhat slower than protection. However, IP-inspired changes in the way transport networks are controlled are rapidly eroding this difference. In this section we will use “protection” to refer to preplanned methods using preassigned capacity and “restoration” for other methods. We will use the term “recovery” when we want to be generic.
110
4.5.3.
John Strand
Recovery Basics
Any method for recovering from a fiber failure requires four fundamental capabilities: (1) Fault detection mechanism to know when there is a problem and what connections have failed; (2) Spare capacity to replace the failed capacity; (3) Switchfabric to give the failed connections access to the spare capacity; and (4) Control logic to trigger the recovery and reroute the failed connections on the spare capacity. In addition, many recovery methods also require afault localization mechanism to determine what links should be avoided in the rerouting process. Consider Fig. 3.30. In this figure, a connection routed X-A-B-Y fails because of the failure of link A-B. Fault detection could either be done at the link level at A or B or it could be done at the ends of the connection (X or Y). To restore the connection, spare capacity on some alternative path bypassing the failure (A-C-D-B in the figure) is needed. To reestablish the connection, A and B need switch fabrics so the new X-A-C-D-B-Y path can be established. Where this switching capability needs to be depends on the location of the spare capacity. Finally, there needs to be a control capability. This could be located in the NEs at A and B, at the ends of the connection (X and Y), or in some network management system connected to A and B by reliable data links. A key criterion for evaluating recovery methods is the amount of spare capacity required. Network topology puts a lower bound on this, as shown in Fig. 3.31. In the figure, the “degree” of a node is the number of links incident on it. In each case shown, we assume that the link from A to Z fails, resulting in a need to reroute the A to Z traffic. In the degree 2 case there is only one possible restoration path, which is to go clockwise from A to Z. In this case, to restore all the A to Z traffic there must be a 100% overbuild, that is, as much spare capacity on each link as there is A to Z traffic to restore. In the
Fig. 3.30 Basic recovery capabilities.
3. Optical Network Architecture Evolution
111
Z
Degree 2 Nodes 100% Overbuild
Degree 3 Nodes 50% Overbuild
Degree N Nodes l/(N-1) Overbuild
Fig. 3.31 Effect of topology on restoration capacity.
second (degree 3) case, there are two potential restoration paths from A to Z. Therefore, if there is a 50% overbuild on each restoration path, all the A-Z traffic can be restored. In general, in a network with all degree N nodes a 1/(N - 1) overbuild is required (last figure). 4.5.4.
Taxonomy of Recovery Methods
Recovery can be done in many different ways, many of them proprietary to either a vendor or a network operator. It also can be done at different levelsservice, SONET, or optical. When confronted with a new method, it is useful to determine how it provides the four basic capabilities listed previously: (1) Fault detection. Is the fault localized before the recovery process starts or not? Localizing the failure can take time and be complex, but it allows the process to bypass only the failed elements, and thus may require less spare capacity. Without fault detection, the connection must be rerouted onto a diverse path end-to-end. (2) Spare capacity. 0 Is the capacity used for restoration dedicated to individual connections or is it shared? If each connection to be restored has dedicated restoration capacity, the restoration process can be very fast and the control logic can be quite simple; however, much more restoration capacity is likely to be needed. 0 Is the spare capacity diverse from the working capacity? If so, protection against a wider range of failures is provided; however, this comes at the cost of having to plan and coordinate across a wider geographical area involving more network elements. ( 3 ) Switchfabric. Are connections restored individually (this is normally called “path” restoration) or are connections restored in bulk (“link” restoration). Also of interest is where the switching is done.
John Strand
112
(4) Control logic. 0
0
0
4.5.5.
Is the network partitioned in multiple recovery domains with recovery controlled independently in each domain or is control global? Recovery time and control process complexity usually both increase at least linearly with network size, which suggests partitioning. Balancing this is the problem of restoring “seam” failures (failures of the links connecting two domains). In addition, if the capacity available for recovery is partitioned so spare in domain A is not available to domain B, additional spare may be needed. Within each domain, is the control process distributed down to the network element level or is it centralized? A centralized process is a potential bottleneck that may make “scaling”22dficult, and it also is a potential single point of failure. On the other hand, it is extremely difficult to test and debug a distributed process. Also, in-service software bugs in distributed systems may be harder to deal with. There have been several spectacular, long-lasting network-wide failures in such systems in recent years. Is the recovery process preplanned for each specific possible failure or is it determined dynamically at the time of failure? Preplanning may speed up the recovery process significantly. However, each time the network state is changed by the addition or removal of a connection, or the failure or placing in a maintenance state of some of the recovery capacity pool, it may be necessary to replan. Also, it is difficult, if not impossible, to have a contingency plan for each possible combination of failure events.
SONETISDH Recovery Methods
Optical Layer recovery methods at this point are mostly proprietary and largely derived fairly directly from the standard SONET/SDH methods. We will therefore first discuss the most important SONET layer methods, and then discuss Optical Layer differences and issues. SONET techniques that aren’t relevant to the Optical Layer are not covered. More detail on these methods may be found in [18], [79], and [99]. Table 3.5 summarizes the key SONET recovery mechanisms in terms of the capabilities described in the previous section. 1 : 1, M :N , and 1 + 1 automatic protection switching configurations are normally used to handle line card, link, and laser failures. They are especially
“Scaling” refers to the efficiency of a control process as the network size increases.
Table 3.5 SONET Recovery Methods Fault Localization Needed
Dedicated Recovery Capacity
Typical Location of Switching Function
Diverse Recovery Path or Capacity Link
M:Nand l+lAPS
No
Yes
Ubiquitous
No
Path-switched rings
No
Possible
ADM line card
Line-switched rings
Yes
No
Mesh
No
Possible
Multipre Recovery Domains
Control within Preplanned Each Domain or Dynamic
Path
Yes
Distributed
Preplanned
Yes
Path
Yes
Distributed
Prcplanncd
ADM
Yes
Link
Yes
Distributed
Preplanned
DCS
Yes
Path
Possible
Centralized or distributed
Either
114
John Strand Switch
I Switch (a) 1 : 1 APS
SWI
(b) M: N APS
(c) 1+1 APS
Fig. 3.32 Automatic protection switching options.
useful for protecting intraoffice connections. When used in interoffice communications, the service and protection capacity is normally transported together in the same cable. These methods are illustrated in Fig. 3.32. In each case, the first number indicates the number of protection channels, and the second number indicates the number of service (working) channels. Figure 3.32b is a generalization of Fig. 3.32a where there are M protection channels for N service channels ( M < N ) . It allows any M of the service channels to be restored simultaneously. 1 :N has been used much more extensively than configurations with M > 1. In Fig. 3.32c, two copies of the signal are sent and a selector at the receiver chooses the better. SONET 1 : 1 and 1 : N APS use in-band signaling (the K1/K2 overhead bytes in the SONET overhead) for coordination. 1 1 APS does not require any signaling, which is a simplification. The price paid for this is the constant use of the protection channel; in the other architectures it is possible to put preemptible connection^^^ on this channel. These could be used for low-priority services or they could be part of the restoration capacity pool for a back-up restoration scheme to be used if APS alone was inadequate. The 1 : 1, 1 :N , and 1 1 configurations have been standardized for both SDH and SONET. To the best of the author’s knowledge, M : N with M > 1 has not been standardized. The path-switched ring configuration is called “SDH subnetwork connection protection” by the ITU [79]. It differs from the 1 : 1 and 1 1configurations just described in two respects: (1) The protection capacity is normally routed diversely from the service connection, and (2) it operates at the path (endto-end) layer. Figure 3.33 gives an example of the most common form, a “Unidirectional Path-Switched Ring” (UPSR). Figure 3.33a shows the underlying network structure: four nodes connected by OC-48s. Each OC-48 is represented by a pair of shaded lines. These represent the two unidirectional transmission paths of the OC-48, with the outside one going clockwise and the inner one counterclockwise. Figure 3.33b shows a lower speed (OC-3 or OC-12) UPSR between nodes 1 and 3 that is routed on these OC-48s. Both the
+
+
+
23
Called “extra traffic” in SONET.
3. Optical Network Architecture Evolution
(a) 4 Node OC-48 Ring
(b) UPSR - Normal State
115
(c) UPSR - Failure State
Fig. 3.33 Path-switched ring (OC-48-based UPSR example).
Fig. 3.34 Generic path-switched ring.
1-3 direction and the 3-1 direction are routed clockwise, on the outer transmission path. Thus the two directions are routed separately. At nodes 1 and 3 there is path termination equipment for the UPSR. Figure 3 . 3 3 ~shows that if the service path from 4 to 1 fails, the 3-1 direction of the UPSR is rerouted onto the counterclockwise loop, thus restoring the connection. The switching is done at nodes 1 and 3, with no involvement by the other nodes. Only the failed direction (3-1) is rerouted. There could be multiple UPSRs between different nodes riding on the same set of OC-48s. More general connecting networks are possible, as indicated in Fig. 3.34. Here the SONET network “cloud” could be composed of a mixture of SONET rings and linear structures. The protection switching is done at the terminations of the path. A line-switched ring configuration was discussed in conjunction with Fig. 3.2 in Section 2.1. All of the recovery mechanisms described so far require that the network be divided into recovery domains with either ring or linear topologies4egree 2 topologies in Fig. 3.31. Planning and operating a network so divided is complex and frequently expensive, as we shall see shortly. Mesh recovery works in an arbitrary topology (degreeN in Fig. 3.31) without having to deal with the planning and operational complexities of a multi-recovery-domain environment.
116
John Strand
Unfortunately, this flexibility makes it virtually impossible to standardize a mesh restoration method to the level that has been done with SONET rings. As a result, all mesh implementations are proprietary. One mesh implementation that has been described in the public literature is AT&T’s FAST Automated Restoration (FASTAR@)[20]. Originallydesigned for a pre-SONET network, it has been extended for SONET. We will briefly describe how it works to give a flavor of mesh restoration. FASTAR’s physical infrastructure consists of Restoration Network Controllers (RNCs)in each office; a centralized restoration controller [theRestoration and Provisioning Integrated Design (RAPID) system]; and a redundant data network to connect them together, and also to connect RAPID to the DCSs, which do the actual rerouting. When a failure occurs, each RNC collects alarms from the affected NEs in its office. After waiting for a brief period to allow all alarms to come in, and also to see if the problem is a transient, it then sends the alarms to RAPID. The RAPID database contains information on all spare capacity available for restoration.24When the first alarm arrives, RAPID opens up an “event window.” While the window is open,25alarms are collected. After the window is closed, RAPID: 0 0
0
Determines what spare capacity has survived the failure. Orders all the connections reported failed according to their restoration priority. Processes each connection in turn. For each, an alternate path with spare capacity is computed, the spare capacity is committed, and commands are sent to the appropriate DCSs to implement the reroute.
FASTAR’s restoration speed is gated by the rate at which the DCS can do reroutes.26 As indicated in Table 3.5, there are many dimensionsin which mesh methods may vary: 0
Recovery actions may be determined dynamically after the failure (like FASTAR) or can be preplanned. A number of levels of preplanning are
24 FASTAR depends critically on the timeliness and accuracy of this database. Keeping a centralized view of a network composed of many types of equipment and distributed around hundreds of nodes and links sufficiently accurate is probably the most difficult challenge a centralized-mesh-restoration developer faces. 25 It is closed when either a specified period elapses with no additional alarms or when a time specifyingthe maximum window period expires. 26 For a U.S. government description of FASTAR, including some restoration times, see h t t p : / l ~ . n c s g o v / n 5 _ h p / I n f o r m a t i o n A s s u r a S e chtm. 3.
3. Optical Network Architecture Evolution
0
0
117
possible. For example, the reroute paths may be preplanned but the specific capacity to be used along the path for a specific connection may be determined dynamically. Control may be centralized (like FASTAR) or may be distributed to the NEs. Rerouting may be done end-to-end (which is allowed by FASTAR) or constrained to stay near the site of the failure.
Many of these alternatives are being vigorously pursued for Optical Layer restoration. We will discuss distributed mesh algorithms in Section 5. 4.5.6.
Ring/Mesh Comparison
ITU and SONET standards specify that restoration should be completed within 50 ms in most cases,27and 200 ms is an achievable upper bound for ring restoration. There are new mesh restoration schemes proposed for the optical layer, and also for IP-based networks, that may be competitive, but these are still mostly theoretical results. Centralized mesh algorithms are likely to take a minute or more to recover from a large failure. Rings are required to have uniform capacity on each of their links. This can cause difficulties, as is illustrated in Fig. 3.35. Figure 3.35a shows a topological structure common in intercity networks-a set of paralleling north-south and east-west fiber routes. If there is a roughly equal amount of demand on each link, then the same amount of capacity is needed in the upper (e.g., A-B-C-D) and the lower (C-D-E-F) “window panes.” But this leads to double capacity on the middle route. Figure 3.35b shows another example of the complexities introduced by the equal capacity constraint. Here, each dashed line shows the routing of some connections, each of which requires more than 50% of the capacity of a ring.
I I
Demands (25 DS3 Each): A-E F-L H-G H-L
(a) Window Pane Network
I I
Shortest Distance Routing Optimized Routing (b) Effect Of Routing On Ring Efficiency
Fig. 3.35 Ring examples.
*’
If a ring is more than 1200km in circumference, or if it is carrying extra traffic, an extra 100ms or so may be required.
118
John Strand
The two networks differ only in how the demands are routed. If each demand is routed on the shortest path (left), the lower loop (A-F-G-H-J-K) requires two rings because of the demand between G and H. The top two loops require a total of three rings because of the demands between E and F. Thus a total of five rings are needed. If the demands are routed as a group with an eye to minimizing the number of rings required (right), only three rings are required. To accomplish this, the demands A-E, H-L, and G-H have been rerouted onto circuitous paths. Routing algorithms sophisticated enough to achieve the optimized routing need either to have an accurate view of future pointto-point demands or the ability to continually do massive rearrangements to preserve optimality as new demands appear, neither of which is practical at this time. A number of studies of the relative economics of ring versus mesh have been done. In [15] it is stated that mesh is “20% to 60% more efficient” than rings. Studies at AT&T of representative national topologies and demand sets have yielded larger differentials-at least 50% more capacity required even if routing is optimized (as in the right routing in Fig. 3.35b), over 100% if it is not. RHK has estimated that ring restoration is 80% more expensive than mesh for the optical layer. A comparison of optical ring versus optical mesh for a very large future IP network also showed significant backbone savings for mesh over rings [105]. 4.5.7.
Virtual Rings
Each SONET line-switched ring has a dedicated NE (an ADM) at each of its nodes. This is acceptable as long as the number of rings in an office is small. The unprecedented growth rates experienced in recent years, however, has raised the specter of offices with hundreds or even thousands of ADMs. Consider the situation shown in Fig. 3.36. Figure 3.36b illustrates the situation at office A using ADMs. All connections wishing to switch rings or terminate in office A must traverse cabling from the ADMs [the small squares in (b)] to the DCS (diamond in the center), which becomes a bottleneck. Furthermore, each
(a) Grid Network
(b) ADM’s & DCS At Office A
Fig. 3.36 Virtual rings.
(c) DCS With Virtual Rings
3. Optical Network Architecture Evolution
119
ADM is physically cabled on the network side to specific physical facilities; recabling would be needed to change this. Figure 3 . 3 6 ~illustrates what is called a “virtual ring.” Here, a single DCS switch fabric replaces all the ADM fabrics. Any two DCS ports may be associated together through the fabric to form the functional equivalent of an ADM. Ring interconnects also are made through this fabric and the line-switched ring-protection algorithm is implemented in the DCS control software. This has major advantages: (1) Intraoffice connections between rings no longer require intraoffice cabling to and from the DCS; (2) any two high-speed ports on the DCS can be made into a logical ADM via a software command; and (3) it is now physically possible for two virtual rings to share protection capacity. Proprietary virtual ring implementations have been made by a number of DCS and OLXC vendors. 4.5.8.
Diversity
For a ring to work properly, it is necessary that the links forming it are physically diverse from each other, otherwise a single failure might cut it in two places and cause a total failure. To determine whether two lightpath routings are diverse, it is necessary to identify single points of failure in the interoffice plant. To do so, we will use the following terms: AJiber cabZeis a uniform group of fibers contained in a sheath. An OTrS will occupy fibers in a sequence of fiber cables. Each fiber cable will be placed in a sequence of conduits-buried honeycomb structures through which fiber cables may be pulled-or buried in a right ofway (ROW). It is worth noting that for economic reasons, ROWs are frequently obtained from railroads, pipeline companies, or thruways. Often several carriers may lease ROWs from the same source; this makes it common to have a number of carriers’ fiber cables in close proximity to each other. Similarly, in a metropolitan network, several carriers might lease duct space in the same RBOC conduit. As discussed in Section 3.3, carriers also trade capacity. In each of these situations, there is the opportunity for a single event to disrupt more than one carrier’s facilities simultaneously.Thus, relying on carriers to be diverse from each other without detailed evaluation of the physical routings is chancy. In a typical U.S. intercity facility network there might be on the order of lo2 major offices. To accurately capture diversity, however, a network with an order of magnitude more nodes is required. In addition to Optical Amplifier (OA) sites, these additional nodes include: 0 0 0
Places where fiber cables enterneave a conduit or right of way; Locations where fiber cables cross; Locations where fiber splices are used to interchange fibers between fiber cables.
John Strand
120 B-A
(a) Fiber Cable Topology
(b) Right-Of-WayKonduit Topology
Fig. 3.37 Fiber cable vs ROW topologies.
An example of the first is shown in Fig. 3.37. Here, the A-B fiber cable would be physically routed A-X-Y-B, and the C-D cable would be physically routed C-X-Y-D. This topology might arise because of some physical bottleneck: X-Y might be the Lincoln Tunnel connecting Manhattan and New Jersey, for example, or the Bay Bridge. The imminent deployment of ultra-long Optical Transport Systems introduces a further complexity: Two OTrSs could interact a number of times. For example a New York-Atlanta OTrS and a Philadelphia-Orlando OTrS might ride on the same ROW for x miles in Maryland and then again for y miles in Georgia. They might also cross at Raleigh or some other intermediate node without sharing right of way. 4.5.9. Unique Features of Optical Recovery In most respects, Optical Layer recovery options and considerations are the same as those encountered in SONETBDH. This should not be surprising because both are multiplexed, connection-oriented networks. There are, however, a few sources of difference, which we now discuss. 4.5.9.1. Opaque Optical Networks
In an opaque OTN, and also when protection is provided for the connectivity between domains of transparency, there is no real difference between the optical situation and that encountered in traditional SONET/SDH networks. 4.5.9.2. Domains of Transparency
Recovery options within a domain of transparency are limited in several ways: 0
Transmission constraints may eliminate some recovery options. For example, in Fig. 3.30 the recovery path (X-A-C-D-B-Y) could be significantly longer than the initial path (X-A-B-Y), so it might have unacceptable noise or other impairments.
3. Optical Network Architecture Evolution 0
121
If wavelength translation options are limited, the spare capacity that can be used to restore a particular connection will be limited. In the extreme, if there is no wavelength conversion capability at all, then the pool of recovery capacity fragments into separate pools for each frequency.
These limitations are much less of a problem in metro networks or other situations where path lengths are limited. 4.5.9.3. Flexible Groupings
SONET line- and path-switched rings normally switch the equivalent of optical channels (e.g., Figs. 3.2 and 3.33). Optical switching has the property that it is much less sensitive to the composition of the optical signal being switched; switching technologies such as MEMS could switch an aggregate multi-OCh signal as easily as they can switch a single OCh. This provides additional implementation options for recovery mechanisms. For example, PADMs can do the equivalent of SONET line switching in two ways, as shown in Fig. 3.38. Figure 3.38 shows a two-fiber PADM. If there is a failure just to the right of it, switching at the OMS level is represented by the loopback labeled “1”; at the OCh level, by “2.” Both options have the effect of looping the entire signal back from the service (S) to the protection (P) fiber. If the ring is implemented in a cross-connect rather than a PADM, Fig. 3.9 shows the equivalent options: OMS switching would be done by a fiber crossconnect (Option C in the figure), whereas OCh switching would be done by the switch labeled “Option B.” If the wavelength bundles called “adaptation groupings” in Section 2.2 (see Fig. 3.7) are being used, the flexibility of optical switching would allow protection switching to be done at this level also. 4.5.9.4. OMS Level Protection Switching
This option would have a number of potentially attractive features: (1) Many all-optical switching technologies, such as MEMS, could switch an aggregate multi-OCh signal as easily as they can switch a single OCh. This should lead to
P
4
P
..
*. *
S
Fig. 3.38 Photonic ADM protection switching options.
S
122
John Strand
economic savings. (2) Since there are fewer OMS than OCh, OMS restoration may scale better than OCh restoration. (3) Wavelength conversion is not an issue. However, OMS restoration can only be done within a domain of transparency. Thus all the issues associated with transparency discussed in earlier sections,28particularly transmission constraints, multivendor interoperability, and problems associated with the introduction of new technologies arise here. Therefore, most attention is currently focused on OCh-level restoration. However, OMS restoration can be effectivein situations where these issues are not compelling, particularly in metro and access networks and when integrated into a single-vendorall-opticalnetworking product such as the ultra-long-haul systems discussed in Section 2.2. OMS protection switching does not protect against the failure of an individual signal before they are wavelength multiplexed. To gain this capability, it must be combined with an OCh-level mechanism, such as 1 :N protection switching, which is discussed next. 4.5.9.5. 1 : N A P S Protection Switching
Providing equipment protection for wavelength-specificcomponents such as a transponder that is transmitting into a domain of transparency can be done in a number of ways. The setting is shown in Fig. 3.39. As shown in Fig. 3.39, an incoming signal (from the left) is fed to a working transponder (with output wavelength hl in the example) and also to a protection transponder (at bottom). Not shown are up to N - 1other incoming signals that are sharing the same protection transponder. Two of the implementation approaches described in [151 are: Transponders
Fig. 3.39
28
1 :N APS protection setting.
Sections 2.3 and 4.2 in particular.
3. Optical Network Architecture Evolution
123
Fixed spare. In this case, Aspare is some other fixed wavelength AN+^. At
0
the Demux point there would need to be an appropriate receiver. Upon failure of a working transponder, the incoming signal is routed by the 1 x N switch to the protection transponder. In addition, to coordinate the ends a fast signaling protocol like that used for SONET, APS would be needed. If the domain of transparency in the center is a large mesh, this protocol could get very complex. In addition, a wavelength is dedicated for protection purposes only. Tunable spare. In this case, the spare transponder is built around a tunable laser. Upon failure this laser would be tuned to the frequency of the failed transponder. In this case, neither the dedicated protection wavelength nor the extra receiver is needed, and no coordination is needed, as nothing in the network has to change when the spare laser is tuned to the failed frequency. The trade-off is the requirement for a tunable laser, which at present would be expensive and present other difficulties.
4.5.10. Multilayer Considerations Recovery mechanisms are available in the service layers, especially for services built on IP. Whenever there are multiple recovery mechanisms that might be invoked by the same failure, coordination of some sort is needed. If this is not done, problems of the sort illustrated by the following time-line are likely (Fig. 3.40). The Optical Layer is at the bottom, the Service Layer above, and time runs from left to right. A link failure is shown at the left, which is remedied through optical protection in a few milliseconds; however, before this was accomplished an alarm was detected by the Service Layer. This triggers many service layer reroutes, a process taking 10s of seconds in a large IP network. These reroutes could also cause user-visible service glitches. Rediscovery of the link is accomplished in IP by another protocol, and could take some time.
IP-Based Service Layer
Optical Layer
10s seconds,
.. ..
.. ..
10s seconds
,
Fig. 3.40 Counterproductive inter-layer recovery behavior.
124
John Strand
When this finally happens, another period of service instability will ensue as the Service Layer reroutes are undone. In addition to these effects, uncoordinated multilayer restoration is likely to lead to excessivecapacity being provided for restoration. The problem here is that it is very difficult for layers to share this capacity; hence if both layers are provided with capacity to handle the same failure scenario, much of this expenditure is wasted. It is thus important both for performance and economic reasons to apply the restoration capabilities of each layer to only the types of failures for which it is most suited. The remainder of this section tries to apply this principal to the Optical Layer and the IP Services layers. The Optical Layer strengths in this respect are: 0
0
0
Line-curd costs. As discussed in Section 4.4,OLXC costs per unit capacity are expected to be significantlylower than those of an IP router. Multiservice support. As long as there are multiple client services, a single restoration mechanism at the Optical Layer reduces the need for separate mechanisms with their associated capacity requirements and operational support costs. Bundle size. The Optical Layer can restore in much larger bundles than can higher layers. Larger bundles implies fewer recovery reroutes need to be done, which should translate into faster overall recovery times.
Relying on the IP layer for recovery has advantages: 0
0
Much finer granularity restoration is possible. This means that “best effort” traffic (which comprises a large proportion of total Internet traffic) can be left unrestored, with corresponding restoration capacity savings. IP layer restoration capacity may be provided to deal with router failures. If this capability is in place, the incremental capacity and complexity required to deal with all failures may be modest.
Doverspike et ul. [l 1, 851 have modeled a number of OLXC and IP-based architectures to study the economic trade-offs more closely. Their conclusion [l 11: “. .. generally, for single fiber failure, OLXC restoration tends to be less expensive than IP Layer restoration except when low fractions of IP traffic need restoration and the OLXC layer is omitted or, possibly, when we need to provide extra capacity to protect against IP layer node (router) failures.” Rather than viewing the choice of restoration layer as an either/or choice, it is likely to be more profitable to look for hybrid strategies that make use of the
3. Optical Network Architecture Evolution
125
strengths of each layer. Gerstel and Ramaswami [15] have identified several cooperative strategies: Optical layer routing to enhance IP layer restoration. a. At its simplest level, this requires the OL to ensure that IP service and protection capacity is kept physically diverse; more generally it requires that the IP layer understand its redundancy constraints and communicate these constraints to the OL when requesting new capacity. (See [19], [37], and [lo81 for more detail on diverse routing.) b. Additional capacity efficiency can be obtained by treating some of the IP layer protection capacity as preemptible “extra traffic” connections in the OL. Careful design is required to ensure that these protection resources will be available when needed. a Multilayerprotection. Recovery responsibility can be divided between layers in various ways. This requires an escalation strategy defining how layers interwork to provide recovery. Some of the building blocks for cooperation are: a. Hold-oftimes. The IP layer could allow a short interval for the OL to execute its procedures before starting its own procedures. b. Controlplane coordination. As will be discussed in Section 5, a new OL control mechanism based on IP protocols is being defined. Common protocols should facilitate interlayer communication. c. Assign responsibilityfor each type offailure to a specijk layer. As a strawman, the OL might deal with single route failures while the IP layer deals with node failures of any type. Intraoffice (tie cable) failures can be efficiently dealt with in the OL using 1 1 APS. 0 Segregate protected and unprotected trafic. The IP layer could separate traffic requiring restoration onto specific OL connections and identify these connections to the OL. This would allow the OL to provide recovery capacity for these connections only. a
+
We have mentioned strategies that use IP layer functionality to reroute around optical layer failures. It also may be possible to use the Optical Layer to recover from IP layer failures in some cases. We give one example here.29 In a large ISP central office, there are typically at least two backbone routers for redundancy in case of router hardware or software failures and to simplify software upgrades. These routers aggregate all the traffic to or from all the provider edge routers that connect to the customers. Figure 3.41 gives an example of such an IP network, with detailed office architecture shown for office B only. 29 See [lo21 for more details. To the author’s knowledge, this scheme has not yet been implemented by any vendor.
126
John Strand Edge
Fig. 3.41 Joint IPlOptical Layer router recovery scenario.
The dotted lines show Optical Layer connections between one of these , other routers, R B I ,and a router at office A and also between RBI and R B ~the backbone router at B. With current IP rerouting, when router RBIfails, traffic from office A to B needs to go around D, E, F, and C to reach office B via R B ~Similarly, . traffic from office A to office C, which originally went through office B, now needs to go around D, E, and F to reach C . Additional capacity may therefore be needed on all these links. If router RB1 fails, it brings down both interoffice link RA-RBI and intraoffice . bandwidth these links occupy is unproductive for the duralink R B I - R B ~The tion of the router failure. This capacity could be reassembled by OLXCBinto a connection from RA directly to Rsz, thus avoiding the need for additional IP layer bandwidth to deal with this failure. Implementing this hybrid approach to router failure requires control plane changes in both the IP and the Optical Layers. Control planes are the topic of the next section.
5. Transport Control Plane In Fig. 3.22 a layered model of the transport signaling and control infrastructure was introduced. This infrastructure was divided into two parts: a “control plane” responsible for real-time control and fault management and a “management plane” providing the network operator with network visibility and control in a static control environment characterized by scheduled activities and delayed response to network state changes.
3. Optical Network Architecture Evolution
127
5.1. TRADITIONAL TRANSPORT NETWORK MANAGEMENT AND CONTROL Legacy transport telecommunications services have evolved to be paragons of reliability, providing guaranteed quality of service regardless of network load. Much effort was directed at optimizing the use of what was then a very expensive resource, bandwidth. All this was accomplished in an era when intelligence could only be provided by centralized Operations Systems (0%). Communications between them was normally by faxed “work orders.” In this environment, microprocessors first appeared in discrete, intelligent NEs communicating with the external world through ASCII-based “craft terminals.” Because intelligence was largely isolated, there was little value in conforming to standards; each network operator developed proprietary OSs, and the Element Management Systems (EMSs) embedded in NEs were largely vendor proprietary. The architecture is illustrated in Fig. 3.42.30 Two carriers, X and Y, are shown in Fig. 3.42. Within each are separate vendor “clouds,” each with one or more vendor-supplied EMSs that communicate with the individual NEs. The EMSs also communicate with a carrier-proprietary NMS (which actually is likely to be a large number of cooperating Operations Systems (OSs), each dedicated to a specific task like restoration, provisioning, or maintenance, and sharing network state information to some extent). Protocols like Common Management Information Protocol (CMIP), Simple Network Management Protocol (SNMP), and Common Object Request Broker Architecture (CORBA) have brought some level of standardization to these interfaces. Human operators track network status and apply controls through both the NMS and the EMS. Intercarrier
______ /--------.
.--_--_________---NE: Network Element EMS: Element Management System NMS: Network Management System
Fig. 3.42 Traditional transport control structure.
30
See [86]for the next level of detail.
128
John Strand
management communications is likely to be between the human operators rather than between system^.^' There is a nontrivial control plane in this structure. Each node in a SONET ring, for example, has state knowledge about the ring, and the nodes cooperate to recover from failures. However, the general architecture puts most multinode functionality in the management plane (the EMS and NMS). This structure has some serious deficiencies from a network operator’s perspective: 0
0
0
0
0
0
NMS software typically is large, complex, costly, and requires a major ongoing software development and maintenance effort. The introduction of a new technology, vendor, or service is frequently slowed down significantly by the need to wait for the necessary NMS software changes. Creating a network-wide “database of record” containing accurate and up-to-date state information is greatly complicated by the proliferation of proprietary software at the EMS and NE level. NEs have minimal ability to automatically determine their connectivity to other NEs. This forces reliance on expensive, error-prone manual inputs. State change information may take some time to percolate up to the NMS level. This can cause problems when rapid action is required, for example, when there is a failure. The ad hoc intervendor interfaces make rapid provisioning or recovery very difficult when multiple vendors are involved. Frequently months are required to establish a complex high-capacity connection. Even when only a single vendor is involved, communications problems and data inconsistencies among all the OSs and EMSs involved in a provisioning introduces significant delays and errors.
Together, these deficiencies are a significant ongoing problem for transport network providers. Ever-increasing competition, together with the evermore volatile needs of customers, make costs, long provisioning intervals, and delays in introducing new services and technologies increasingly intolerable.
5.2. EMERGENCE OF ALTERNATIVE CONTROL PLANE ARCHITECTURES The management and control planes for the Internet and for ATM data networks were developed more recently and have avoided many of these problems. 31 For voice services there is of course a rigorously standardized service-controlplane that allows calls to be established and managed across carriers. We are talking only of the transport layers here.
3. Optical Network Architecture Evolution
129
Network elements are assumed to be intelligent and able to support a much “thicker” control plane, and thus require less NMS and EMS functionality. For someone aware of the problems discussed in the previous section, these new control planes have some very attractive features: 0 0
0
0
NEs could automatically discover their neighbors. This information could be disseminated throughout the network, and each node could build its own view of the entire network. As a byproduct, this process would generate much of the accurate, up-to-date state information needed by the NMS. This has come to be called “The Network as the Database.” This information could be used to distribute provisioning, fault localization, and restoration functions to the NEs. In this approach, the necessary software would be provided by the equipment vendor; this spreads software development costs over all the vendor’s customers and reduces the possibility of network operator software development bottlenecks slowing down the introduction of new technologies and services. This software would implement standardized, proven protocols derived from the Internet and ATM worlds. The standardization might allow multivendor interworking and even a degree of multicarrier interworking. Furthermore, public domain or commercially available implementations of the basic protocols are readily available, and programmers familiar with this software are also readily available. If this software proves to be adaptable to the needs of the Optical Layer, this all should greatly speed up the realization of the desired capabilities.
From a carrier’s perspective, then, a vision of an alternative management and control structure has begun to emerge. In this vision, vendor-supplied OLXC control-plane software will automatically do some key NMS functions such as database creation, provisioning, and restoration, thereby allowing a lower-cost, streamlined NMS. Furthermore, the new control plane had the promise of supporting much more rapid provisioning and less painful intervendor and even intercarrier interworking. In the longer term, there is a hope that if the Optical Layer control plane is compatible with that used by the Internet, then additional ways to be responsive to the Internet’s needs will emerge. OLXC vendors, particularly start-ups, were quick to see that they could inexpensively differentiate their hardware products by modifying existing implementations of the basic protocols. Today virtually every OLXC vendor claims to have a control plane based on this approach. Additional support for the use of Internet protocols came from the Internet community. Anticipating the necessity of incorporating optical switching
130
John Strand
if they were to continue to scale to meet exploding demand, they greeted the possibility of common Internet-Optical Transport control protocols with enthusiasm. If the transport paradigm could be changed from one of fixed “pipes” that could only be reconfigured on a time scale of weeks to months to one of the transport network as a sort of giant distributed circuit switch with a dynamically reconfigurable backplane, new dimensions in traffic engineering might emerge. Vendors with substantial Internet equipment expertise were particularly enthusiastic, foreseeing the possibility of gaining a foothold in the massive telecom equipment market. Both established and start-up network operators have shown considerable interest in these developments. The Carrier Working Group within the Optical Interworking Forum (OIF) recently produced an “Optical Services Framework and Associated Requirements” document [43] that provides a good snapshot of current services thinking in the carrier community as it relates to optical networking and software control. They articulated the following objectives for a new control plane.32 a Promote a standardized optical network control plane, with its associated
inter$aces andprotocols. Ensure that intelligent optical networking (OTN) products from different vendors or employing different
a
a
technologies can interwork at the control plane level. This is not meant to preclude vendor specific extensions so long as the extensions do not degrade total network performance. However, it is unacceptable to expect carriers to write software to conceal OL vendor or technology incompatibilities. Provide rapid automatic end-to-end provisioning within the optical network, including routing and signaling by the controlplane. “Endto-end” is an important part of this objective; the value of rapid provisioning in the core network is greatly reduced if traditional provisioning intervals are encountered in metro and access networks. Hence interdomain interworking is viewed as being key to realizing many of the hoped-for benefits. It is recognized that rapid interdomain provisioning must be built upon rapid automatic provisioning within a single domain. In this document “provisioning”refers to network provisioning only and does not necessarily include other customer-facing aspects of provisioning like the establishment of a new account. Provide restoration, diverse routing, and other Quality of Servicefeatures within the control plane on a per-service-path basis. Per-connection
32 The current author was the chair of the OIF group producing this report. It is working text and not an official OIF Technical Report and is not binding on the OIF or its members.
3. Optical Network Architecture Evolution
e
e
e
e
e
e
e
131
control of these capabilities is important because of the anticipated diversity of needs of the OL users. Provide policy-based call acceptance and peering policies and support usage-based accounting based on start and termination time, bandwidth, and other resources requested. These capabilities are necessary to support a pay-for-service business model and otherwise safeguard carrier resources, which carriers expect to be basic to their OL plans. Oger carrier-spec@c “branded” services. A fundamental carrier business need is the ability to put together packages of features, quality of service, support, and pricing, which they can then market. Rapid deployment of new technologies and capabilities with no network service disruption. In particular, deployment of a new vendor X capability should not depend on vendor Y’s willingness to upgrade their control plane software. It is recognized that the part of the network served by vendor Y equipment may not get the benefits of the new capability in this case. Protect the security and reliability of the optical layer, andparticularly the controlplane. The damage, which could be done if this is not accomplished, is beyond measure. Provide the ability for the carrier to control usage of its network resources. The carrier will control what network resources are wailable to individual services or users. Therefore, in the event of capacity shortages this ability will allow the carrier to ensure that critical and priority services get capacity. Reduce the need for carrier-written OS software through heavy use of open protocols and vendor and third-party software, particularly for network provisioning and restoration. Carrier OS software development bottlenecks frequently are on the critical path to technology and service innovation. Improvements in this area are at least as important to many carriers as is the prospect of very rapid provisioning. Note, however, that entire functions need to be off-loaded in most cases; if this is not done the carrier OS function remains, with the added requirement of interfacing with the new control plane software. Ensure the scalability of the Optical Network. Several aspects of scalability are highly important in a carrier’s network: e Node scalability-the size of a single node is expected to be sized anywhere between several hundred and several thousands optical terminations. e Network scalability-the size of an interconnected network is expected to be anywhere from several to hundred or thousands of nodes.
132
John Strand
5.3. INTERNET PROTOCOL BACKGROUND
To understand the proposed control plane, some exposure to a few Internet concepts are necessary. We provide only the barest possible peek into this vast area. Readers interested in learningmore could start with [87,88,89,90, or 911. 53.1. Internal Protocol (IP)
IP transmits blocks of data called packets from sources to destinations, where sources and destinations are hosts identified by fixed-length globally unique addresses. Each packet contains a destination address and is passed through a sequence of switches called “IP routers.” In a router, each packet is processed independently; IP has no concept of a “connection.” Its address is matched against entries in aforwarding table. The matched entry in this table specifies the output port through which the packet should be sent. 5.3.2. Domains There are millions of IP addresses. Indeed, the 32-bit address space supported by the current version of IP (IPv4)is in danger of being exhausted. Furthermore, the Internet is in a constant state of flux. Therefore, it is impossible and undesirable for any node to have an up-to-date entry in its forwarding table for each of these addresses-the signaling and processing overhead would overwhelm the Internet. Consequently,the Internet is divided into many thousands of domains (formally, “Autonomous Systems (ASS)”; also called “Routing Domains”). Each router keeps detailed forwarding table entries for addresses within its domain. Packets for other addresses are sent to a “border router” that communicateswith its peers in other domains and handles the processing of packets entering or leaving the AS for other ASS.Each AS is under the control of a single administrative entity and comprises a “domain of trust” within which state information is freely shared and routing decisionsnot questioned. 5.33. Open Shortest Path First (OSPF)
OSPF is a “routing protocol.” An instance of OSPF33runs in each router, where it constantly communicates with its peers to keep an up-to-date view of the network topology. Whenever there is a change, OSPF modifies the forwarding table, which immediately modifies the way subsequent packets travel through the network. OSPF is an Interior Gateway Protocol (IGP), which means that it handles the routing within a single domain only.
33
Or an equivalent protocol lie IS-IS. Within an AS a single IGP needs to be used.
3. Optical Network Architecture Evolution
133
5.3.4. Link State Advertisements (LSAs) OSPF routers send LSAs to all their peers in their domain. These define the “linksYy-onnections between adjoining routers. With this information, each router can maintain a complete and up-to-date picture (Link-State Database) of the global topology of the entire domain. It then runs an algorithm (a “Shortest Path First” algorithm, whence the name OSPF) that determines how to construct the forwarding table. Each time an LSA update is received, the algorithm is rerun.
5.3.5. Neighbor Discovery and Maintenance Each OSPF router periodically (by default, every 10 seconds) sends a “Hello” message out of each of its ports. It learns of the existence of a neighboring router when it receives the neighbor’s OSPF “Hello” in turn. If too much time elapses between receipt of these messages (nominally 40 seconds), the router will stop advertising the unresponsive link and also will start routing packets around the failure. This time dominates “convergence time”-the time it takes all the forwarding tables in a domain to stabilize after a failure.
5.3.6. Border Gateway Protocol (BGP) BGP4 is the interdomain routing protocol (commonly called an “Exterior Gateway Protocol”) currently used over the Internet. BGP nodes (called “speakers”) exchange summarized information about the addresses they can reach and the sequence of ASSthrough which the address would be reached. Trust between ASS is not assumed; each BGP speaker applies “policy-based controls” to determine which ASSit wishes to peer with and what traffic it is willing to accept from each peer. BGP does not do neighbor discovery; instead, peering relationships are manually configured. Unlike OSPF, where the onginating router did its routing based on a global-link-state database, each BGP speaker keeps a summary list of distant addresses and the distance to each through each adjacent peer (the “distance vector”). Distance is normally measured as a hop count. Then a speaker will normally route packets to a given destination through the peer that had advertised the fewest number of hops. 5.3.7.
Multi Protocol Label Switching (MPLS)
IP is a connectionlessprotocol, meaning that each packet is routed independently; thus if a block of data is broken up into multiple packets they might be routed independently. An analogy has been drawn to copying a novel onto a set of postcards. MPLS creates the equivalent of connections (called “Label Switched Paths,” or LSPs) by attaching an additional label to each packet in a flow. MPLS label processing is illustrated in Fig. 3.43. In Fig. 3.43, a packet with IP address X is shown entering from the left. A table look-up in the first MPLS node results in the label “6” being appended
I 1: 110; 6 11 ;II Label
I
Address
Label Interface
X
In I Y
IP
7
O0
n 0 MPLS Ingress Router
MPLS Node
Fig. 3.43 MPLS label processing example.
MPLS Egress Router
3. Optical Network Architecture Evolution
135
at the front of the packet and the out interface 0 being used for the packet. In the second MPLS node the label 5 replaces the initial label and out interface 3 is chosen. In the final node, the MPLS label is “popped” off to reveal the initial packet, which is then routed over out interface 0. As long as the MPLS tables are unchanged, all packets with the same destination address that enter this ingress router will be routed in exactly the same fashion. An LSP is a “virtual” connection. Unlike a TDM connection, creating an LSP does not commit bandwidth or use bandwidth when idle. MPLS can make use of this feature to set up numerous LSPs for restoration purposes without consuming bandwidth. To OSPF, an MPLS LSP can be considered as a link in its link state database. MPLS has many other features of importance. One that should be mentioned is the ability to have nested LSPs. This provides a sort of multiplexing capability that is of importance for our application because it will allow an optical connection to be treated as an LSP on which other LSPs are routed. 5.3.8.
Label Distribution Protocol (LDP)
LDP allows MPLS routers to request that an LSP be set up to a specified address. This is done by sending a label request to the desired node. If no problems are encountered, a second message is passed back along the same path, assigning the labels and establishing the LSP.
5.3.9. Constraint-basedRouting Label Distribution Protocol (CR-LDP) This extension of LDP allows the LSP establishment process to be constrained to a specific physical path. It also allows the reservation of resources (bandwidth) along the path. There is an alternative protocol called RSVP (Reservation Protocol) that performs essentially the same function.
5.4. MPkS AND GENERALIZED MPLS (GMPLS) GMPLS (closely related to MPhS, or ‘‘MP Lambda S”) is an extension of MPLS and related protocols to form the basis for a control plane for Optical Networks [24, 49, 921. In fact, it is more ambitious than this: The goal is to support devices that do packet switching and also switching in the time, wavelength, and space dimensions by enhancing the connection-orientedprotocols used by MPLS and its traffic engineering extensions. For Optical Networking, GMPLS: 0
Replaces the MPLS label discussed earlier with the wavelength frequency, which acts as an implicit label, thus treating an OTN
136
0
0
John Strand
connection as an LSP. An OXC is then analogous to an MPLS node and wavelength translation to a label swap. MPLS’s label-nesting capability allows the multiplexing of multiple SONET connections onto a single wavelength to be handled naturally. Uses an extended IGP (e.g., OSPF) to advertise information about the OTN topology and resource availability (e.g., spare capacity). Its routing algorithm then also needs extension or replacement so that it can compute appropriate paths in the OTN. Uses an enhanced CR-LDP or RSVP to set up connections.
GMPLS requires enhancementsto the protocols used in MPLS to deal with peculiarities of OTNs: 0
0
0
0
0
0
0
0
0
Optical bandwidth comes in a few discrete sizes (OC-58, OC-192, etc.), whereas MPLS LSPs can be any size. Optical bandwidth is real and physical, whereas MPLS LSPs are virtual and need not consume any bandwidth if idle. It is not possible to set up standby optical connections for restoration or other purposes without consuming bandwidth. Overbooking optical capacity is not possible; the number of wavelengths on an OTrS is a hard constraint. Optical links have a number of other attributes that affect routing that will need to be advertised, for example, the acceptable formats (SONET, Ethernet) and bit rates (is OC-768 supported on the link). (This is discussed in detail below in Section 5.6.) Optical connections are normally bidirectional while LSPs are unidirectional. Labels are no longer abstract identifiers. They need to be mapped to specific wavelengths or time slots. Fast fault detection and isolation is needed if restoration is to be supported. Unlike normal data applications, control traffic cannot be directly inserted into the user data stream without an OEO conversion and other changes. Thus there needs to be a separate data communications infrastructure for the control plane. All the routing complexities discussed in Section 4,particularly impairments and connectivity constraints, need to be dealt with.
One area where the challenges for GMPLS are likely to be particularly daunting is restoration. This area is discussed in [lo31 in more detail.
3. Optical Network Architecture Evolution
137
5.5. IP-CENTRIC CONTROL PLANE GMPLS is an important element in a new IP-centric control plane, but there are other elements required as well. The overall network framework is shown in Fig. 3.44. GMPLS deals with routing within a single carrier cloud, and thus defines part of the IaDI-the interface between NEs within a carrier domain. It does not define the intercarrier interface (IrDI) or the User-Network Interface (UNI). We turn next to the UNI and alternative ways in which a user's provisioning needs can be communicated to the OTN.
5.5.1. Automated Provisioning Interface Alternatives Some of the principal alternative modes for automated provisioning that are being discussed are illustrated in Fig. 3.45. Figure 3.45a illustrates the two principal models being discussed: 0
The UNI model is client-server. The client requests a connection with certain attributes by sending signaling messages defined by the UNI
UNI. User-Network Interface IaDI. Intra Domain Interface (aka Network-Node Interface) lrDl Inter-Domain Interface (aka Network-NetworkInterface)
n
Fig. 3.44 Network interfaces.
UNI-C
UNI-N
Client Connection (b) UNI Configuration (Client to ONE)
UNI-C Network
UNI-N
Network Client
(a) UNI 8 Peer Provisioning Models ([24])
(c) UNI Configuration (3rd Patty Signaling)
Fig. 3.45 Alternative provisioning models.
John Strand
138
0
standard, and the OTN responds with success and status information. Internal network topology and state information is not disseminated to the client. In the peer model, routers are “peersyyof the ONES inside of the network. Topology information (LSAs) and other state information necessary to do routing is shared. If a router wanted to establish a connection, it would compute the path itself and then send the appropriate CR-LDP/RSVP messages through the OTN to set up the connection.
The peer model was proposed first. It seems to be quite appealing to many Internet people, but has run into major resistance from the carriers because it seemingly does not allow carriers to control their resources34The Optical VPN service concept proposed by the OIF Carriers Group (see Section 3.4 and [43]) is an attempt to meet the need without compromising network integrity. Figures 3.45b and 3.4% give several variants on UNI. Here “UNI-CYy and “ U N I - N represent the UNI client and network processes, respectively. In Fig. 3.45b the client device and the ONE directly interact across the UNI. In Fig. 3.4% the devices are not UNI-capable. Instead, separate client and network management entities negotiate the connection. This may be appropriate when one or both entities are SONET or other legacy equipment. 5.5.2.
Connection Attributes
The proposed control plane would allow fixed-bandwidth connections to be requested, torn down, and modified. To establish a connection, the desired attributes of the connection must be specified. In [93] the major types of attributes to be supported were identified as: 0
0
0
0
IdentiJcation attributes. Source and destination nodes, when the connection is desired. Basic connection type. Framing type (SONET, Ethernet, etc.), bandwidth, and directionality (unidirectional, bidirectional). Priority, preemptibility, protection, and restoration requirements. Routing constraints. This could include explicit routing or diversity constraints.
34 An interested reader might want to compare the peer model with the OIF Carrier Group objectives outlined in Section 5.2. There are a number of discords: It is unclear how policy-based call acceptance or carrier-specific “branded” services would be supported, for example. Security issues and the ability of the carrier to control usage of its own resources also may be troublesome. Thesemay not be show-stoppers,but at the least: refinement of the peer model and/or modification of carrier objectives will be needed.
3. Optical Network Architecture Evolution
139
5.5.3. Neighbor Discovery This refers to the process that determines the port-to-port connectivity of the ONES. Both ONE-ONE connectivity within the OTN and client-ONE connectivityneeds to be determined. Automation of this process is particularly important because of the hundreds or even thousands of fiber connections expected in a single office. There are many types of interfaces currently in use in OTNs that differ in many respects, therefore a variety of methods are required to accomplish this [24]. The basic functions, however, are similar to those discussed in Section 5.3.
5.5.4. Service Discovery Those adjoining ONESand client devices connected to the OTN need to determine what the capabilities of each of their links are, for example, whether both OC-48 and OC-192 are supported on the link.
5.5.5. Topology and Resource Discovery OSPF extensions are needed to determine and globally disseminate information on spare bandwidth, multiplexing capabilities, and protection needed for route computations.
5.5.6. Standards Activities There are four groups involved in standardslimplementationagreement work in the optical arena: 0
0
0
The International TelecommunicationsUnion (ITU),35and particularly its Telecommunication Standardization Sector (ITU-T), which is an international government-sanctioned group that has traditionally controlled telecommunications standards. The ITU-T has a reputation for being thorough, but slow; they are trying to speed up their process. The Internet Engineering Task Force (IETF), and particularly its IP-over-Optical(IPO) working The IETF is the driver for all Internet-related standards and the nexus of IP-related activities. They tend to move rapidly. Before standardizing anything they require two working implementations. Their standards are called “Requests for Comments” (RFCs). Working papers are called “Internet Drafts.” The Optical Domain Services Interconnect (ODSI)37is an informal association of more than 100 companies, primarily equipment vendors.
35 See itu.int. 36 See ietEorghtm.charters/ipo-charter.htm1.
37 See odsi-coalition.com.
John Strand
140
They have developed a functional definition of a mechanism that will allow electrical layer devices to automatically signal the optical network to establish high-speed bandwidth on demand. They anticipate turning their results over to formal standards activities. The Optical Interworking Forum (OIF)38is an informal group with over 300 members, both equipment vendors and carriers. Its goal is to foster the development and deployment of interoperable products and services for data switching and routing using optical networking technologies. The OIF is working on both physical layer and control plane (OIF) documents. The OIF and ODSI are not sanctioned standards groups. Instead, they are trylng to develop “implementation agreements”-specific focused specs that will allow implementation to move onward. The relationships between all these bodies is confusing and there clearly are overlaps. Many people seem to be following the strategy of submitting their standards contributions to multiple bodies in the hope that somehow they will have the desired impact. The proper relationships among all these groups are unclear and controversial. There are some who would like to see all standards work carried out in the ITU. Others would like to see more specialization. This might lead the IETF to be the arena for dealing with protocols, for example, and the ITU the arena for dealing with physical layer standards and for detailing and formalizingthe major standards. The author would like to see the carriers take a more active role in ensuring that the requirements driving the various bodies are consistent. 5.6. ROUTING COMPLEXITIESARISING FROM DOMAINS OF TRQNSPARENCY
Plans for an IP-centric control plane have incorporatedsome of the idiosyncracies of optical technology. However, the work so far is principally relevant to opaque networks with OLXC nodes. This greatly simplifies the control-plane design. In this section we look at some of the additional considerations that arise if we wish to apply this control plane within a domain of transparency. 5.6.1.
Routing Complications
Section 4.2 summarized the state of routing and wavelength assignment in large domains of transparency. Several issues for the control plane arise from this discussion: 0
38
Impairments cannot be ignored in large domains. Additional constraints will need to be imposed on routing. These constraints see oiforum.com.
3. Optical Network Architecture Evolution
0
0
0
0
5.7.
141
depend on additional physical parameters that will need to be determined and advertised, and the specific constraints required in a given situation may depend on the design and engineering of the domain. Port-to-port connectivity across a domain of transparency will be limited by the level of wavelength translation supported within the domain. It will also be constrained by implementation specifics such as adaptation grouping capabilities. At a minimum, significantly more connectivity information will need to be advertised. A number of additional software-controllable components (beyond the OLXC) that can change the internal connectivity of the domain can be expected. These include PADMs, tunable lasers and filters, and dynamic devices that allow changes in the bandwidth assigned to an adaptation grouping and the wavelength spacing within the grouping. At a minimum, information about the connectivity changes resulting whenever these components are reconfigured will need to be advertised. In addition, these components potentially could be configured by the control plane. Because the route selected must have the chosen wavelength available on all links, this information needs to be considered in the routing process. This is discussed in [94], where it is concluded that advertising detailed wavelength availabilities on each link is not likely to scale. Instead they propose an alternative method that probes along a chosen path to determine which wavelengths (if any) are available. This would require a significant addition to the routing logic normally used in OSPF. Choosing a path first and then a wavelength along the path is known to give adequate results in simple topologies such as rings and trees [74]. This does not appear to be true in large mesh networks under realistic provisioning scenarios, however. Instead, significantly better results are achieved if wavelength and route are chosen simultaneously [76]. This approach would, however, also have a significant affect on OSPF.
TRADING OFF THE CONTROL PLANE, SYSTEM DESIGN, AND CARRTER PLANNINGYENGINEERING
As domains of transparency become both larger and more software reconfigurable, these complexities become increasingly of concern. It is important to note that at present this evolution is largely technology driven. Vendors pushing the technology envelope are competing fiercely to provide solutions that have higher capacity, can go further all-optically,are more reconfigurable, and are more cost-effective. As vendors pursue their diverse visions, it is quite plausible that the Optical Layer of the future will be made up of heterogeneous technologies that differ significantly in their control-plane needs.
142
John Strand
Carriers will wish to take advantage of the new technologies; however, they are also going to want to gain the service and operational benefits offered by the new control plane concepts, which are made more difficult to achieve by this evolution. What are the choices? Alternative approaches that deserve consideration are: Control-planesolutions. The control plane could be enhanced in a number of ways: a. All-encompassing intradomainprotocols. GMPLS could attempt to include the full range of technological options and constraints and evolve as these evolve. This will be, at best, a very challenging approach, in this author’s opinion, which will be made more difficult by the need to do periodic in-service software upgrades in a multivendor network. b. Per-domain routing. In this approach, each domain could have its own tuned approach to routing. Interdomain routing would be handled by a multidomain or hierarchical protocol that allowed the hiding of local complexity. Single vendor domains might have proprietary intradomain routing strategies. This option is discussed further in [19], [36], and [37]. System architecture solutions. Vendor system designers could be less aggressivewith respect to nonlinearities, and therefore somewhat suboptimal, in exchange for improved scalability, simplicity, and flexibility in routing and control-plane design. As a hypothetical example, if control-plane protocols did not deal with chromatic dispersion, carriers would require their vendors to provide transport systems engineered to keep this impairment from ever being binding. Transportplanning/transmission engineering solutions. a. Transmission engineers could be required to only deploy domains where every possible route met all constraints not handled explicitly by the control plane, even if the cost penalties were severe. b. At (selected) OLXCs within a domain of transparency, the control plane could insert O/E/Oregeneration into routes with transmission problems. This might make all routes feasible again, but at the cost of additional cost and complexity, and with some loss of rate and format transparency.
The author is not aware of any studies that evaluate these alternatives generically. 5.8. HETEROGENEOUS TECHNOLOGIESAND MULTIPLE DOMAINS
One of the approaches for dealing with the routing and control problems associated with complex domains of transparency that were identified in the
3. Optical Network Architecture Evolution
143
preceding section was to divide a network into multiple routing domains, each with relatively homogeneous technology (e.g., single vendor). Then a multidomain protocol (the optical equivalent to BGP or multiarea OSPF for the Internet) would be used for multidomain interworking. There are a number of other scenarios where multiple routing domains are likely to be needed. Two of these are: e
e
Metro-core interworking. Deployment of intelligent optical networking technology is planned or under way in both intercity (core) and metro portions of a number of carrier networks. It is essential that the metro and core subnetworks be able to interwork as soon as possible if we are to realize the fast provisioning potential of these deployments, because a large proportion of the anticipated connections will need to traverse both metro and core subnets. Multi-carrier interconnection.As optical networking becomes established, opportunities and needs for routing optical connections over multiple competing carrier networks are presented. This is done today in the United States with private lines, many of which are routed through an intercity carrier network and one or more RBOC networks. Internationally also, multiple carriers are often involved. As the number of optical networks supporting rapid provisioning increases, the needs for this sort of interconnection can only increase.
More details on these applicationsand some initial requirementsfor amultidomain optical protocol can be found in [97]. At the time of writing, work to define the necessary protocols is just starting (see, e.g., [96], [98]).
5.9. ALTERiVATIVE CONTROL-PLANE APPROACHES
The distributed, IP-inspired control-plane work described in the last few sections is not the only possible approach to controlling and managing an optical network. In this section some dissenting opinions and alternative approaches will be mentioned. Gerstel [22] argues that it would be a mistake to adopt Internet-style distributed network control. He argues that a telecom-stylenetwork management interface augmented with a minimal control plane and a service-layer interface between management systems would be a better choice. The control plane would be distributed but would be limited to fault management. Connection setup would only be provided for automatic protection purposes. The key functions addressed by the control plane discussed in the previous sections would be performed by network management systems. There would be multiple single-vendor domains, each with its own EMS; they would interoperate
144
John Strand
at the NMS level using a protocol based on the CORBA interface standards developed in the telecom industry. d s l i Hjdmtjrsson and his collaborators [go], on the other hand, argue that a thorough-going integration of the IP and Optical layersis a better architecture. In this architecture, each node integrates a router and a PXC. Intelligence remains entirely in the router, which is responsible for all networkingfunctions, including optical resource management. They argue that collapsing these two layers together would remove a major source of complexity. Their proposal is consistent with a strong form of the GMPLS peer model. Mukherjee [lo51 summarizes an alternative control plane approach for domains of transparency that is not based on each node keeping a global Link-State Database. In this distributed-routingapproach, routes are selected in a distributed fashion without knowledge of the overall network topology. Each node maintains a routing table that specifies the next hop and the cost associated with the shortest path to each destination on a given wavelength. The connection request is routed one hop at a time, with each node along the route independently selecting the next hop. If a node on the path is unable to reserve the desired wavelength on a link, it sends a negative acknowledgment back along the reverse path. Once the destination node is reached, an acknowledgment is sent back along the reverse path; this triggers OLXC configurations as it goes. This approach has the advantage of not requiring distant nodes to be cognizant of impairments in distant parts of the network, but has the disadvantage that it makes constraint-based routing, such as is required for diversity, more complex.
6. Summary We have tried to emphasize the interconnections between technology, network structure, services, and economics. Low cost is critically important, but this does not imply that a vendor can focus strictly on hardware costs. Because most telecommunications costs are in operations rather than in hardware, the maintenance and provisioning costs associated with a design must always be a major consideration. Control-planearchitects ignore the many unique aspects of optical technology at their peril. The unique aspects of IP-based services must be foremost in the minds of everyone. Both network operators and equipment vendors will be struggling to differentiate their offerings. It appears that much of this struggle will focus on software-based fmctionality--"soft optics" and new control and management planes. Of critical importance will be the sorting out of functionality between the Optical and IP-based layers. Where should restoration be done? Who should be responsible for fault detection and management? Can a rapidly reconfigurable optical network compete with Tier 1 ISPs to provide IP connectivity?
3. Optical Network Architecture Evolution
145
In this context, some of the most interestingtechnological developments are those that make Optical Transport Systems more flexible and reconfigurablesoftware-reconfigurable OADMs, tunable lasers and receivers, and the like. They have the potential to turn an OTS into a sort of giant distributed optical cross-connect. Equally important are the developmentsin all-opticalswitching fabrics. As optical technology matures, all-optical “domains of transparency” are likely to become more important. They promise considerable cost savings by reducing the need for expensive transponders, and they may offer additional advantages in the form of better scalability and service flexibility. Complex architectural and economic trade-offs will need to be made to get the right balance of domain diameter, unit cost, and operational complexity. Fault management in all-optical networks is a continuing concern. There is also a tension between optimizing the use of optical technology and unduly complicating the control plane that must determine in real-time how to configure and manage the network. It is likely that this tension will lead to domain-specific control with some sort of distributed or centralized interdomain network management. The future of all-optical domains is complicated by another trend-the economic and operational improvements that can be attained by integrating layers. In one direction, IP vendors may see reasons to tightly couple optical functionality into their routers while keeping the discrete optical layer as simple and “dumb” as possible. In another direction, particularly in metro networks multiservice provisioning, platforms integrating TDM ,optical, and data fabrics into a single box are getting a lot of attention. Both of these trends couple optical network design with other worlds, and therefore may complicate the efforts to optimize the use of optical networking capabilities. Changes in the structure of the industry are also of great importance for the evolution of optical network architecture. Of particular note is the fragmentation of the long-haul transport infrastructure as many new entrants appear. Also important is the structure of the Internet world-its preference for very large bandwidth connections,as well as the emergence of server farms and carrier hotels, which may change the relationship between customers and their network providers and offer new business models, such as bandwidth trading, to get established.
Acronyms ADM ANSI AS ATM BGP
Adddrop multiplexer American National Standards Institute Autonomous system Asynchronous transfer mode (protocol) Border gateway protocol (Internet routing protocol)
146
John Strand
CLEC CRC DCS DPRING EMS EO EOY FEC GUI IaDI IGP ILEC IP IrDI ISP ITU ITU-T LAN LEC LR LSA LSP LSR MAN MPLS MTBF MTTR NE OA OADM OCh OE OEO OH OL OLXC OMS ONE OSPF OTN OTrS OTS
Competitive local exchange carrier Cyclic redundancy code Digital cross-connect system Dedicated protection ring (SDH terminology) Element management system Electrical-to-Optical(conversion) End of year Forward error correction Graphical user interface Intra-domain interface (ITU terminology) Interior gateway protocol Incumbent local exchange carrier Internet protocol (Internet layer 2 protocol) Inter-domain interface (ITU terminology) Internet service provider International telecommunicationunion Telecommunication standardization sector of the ITU Local area network Local exchange carrier Long reach (referringto lasers) Link state advertisement Label switched path (an MPLS connection) Label switched router (MPLS node) Metropolitan area network Multiprotocol label switching Mean time between failures Mean time to repair Network element Optical amplifier Optical add-drop multiplexer (either optical or electrical fabric) Optical channel (ITU terminology) Optical-to-Electrical(conversion) Optical-electrical-optical (as in a regenerator) Overhead (control information added to a packet or frame) Optical layer Optical layer cross-connect (may have optical or electrical fabric) Optical multiplex section (ITU terminology) Optical network element Open shortest path first (an Internet routing protocol) Optical transport network Optical transport system Optical transmission section (ITU terminology)
3. Optical Network Architecture Evolution
PADM PDH PN PXC QOS
RWA SDH SHR SLA SONET SPRING SR TCP VPN WAN
147
Photonic (all-optical) adddrop multiplexer Plesiochronous digital hierarchy (DSO, DS1, DS3) Photonic (all-optical) network Photonic (all-optical) cross-connect Quality of service Routing and wavelength assignment Synchronous digital hierarchy (International ITU analogue of SONET) Self-healingring Service level agreement Synchronous Optical NETwork Shared protection ring Short reach (referring to lasers) Transmission control protocol (principal Internet layer 3 protocol) Virtual private network Wide area network (refers to intercity networks)
References Note: References are primarily to survey and overview material. References to primary research may be found in these articles. [l] R. Ramaswami and K. N. Sivarajan, Optical Networks: A Practical Perspective, San Francisco: Morgan Kaufmann, 1998. [2] B. Mukherjee, Optical CommunicationsNetworks, New York McGraw Hill, 1997. [3] P. Bonenfant and A. Rodriguez-Moral, “Optical Data Networking,” ZEEE CommunicationsMagazine, vol. 38, no. 3, March 2000, pp. 63-70. [4] N. Ghani, S. Dixit, and T-S. Wang, “On IP-over-WDM Integration,” IEEE CommunicationsMagazine, vol. 38, no. 3, March 2000, pp. 72-84. [5] J. M. H. Elmirghani and H. T. Mouftah, “All-OpticalWavelength Conversion: Technologies and Applications in DWDM Networks,” IEEE Communications Magazine, vol. 38, no. 3, March 2000, pp. 86-92. [6] 0. Gerstel and R. Ramaswami, “Optical Layer Survivability: A Services Perspective,” ZEEE Communications Magazine, vol. 38, no. 3, March 2000, pp. 104-113. [7] J. M. H. Elmirghani and H. T. Mouftah, “Technologies and Architectures for Scalable Dense WDM Networks,” ZEEE Communications Magazine, vol. 38, no. 2, Feb. 2000, pp. 58-66. [8] S. Yao, B. Mukherjee, and S. Dixit, “Advances in Photonic Packet Switching: An Overview,” IEEE CommunicationsMagazine, vol. 38, no. 2, Feb. 2000, pp. 84-94. [9] D. Cavendish, “Evolution of Optical Transport Technologies: From SONETI SDH to WDM,” IEEE CommunicationsMagazine, June 2000, pp. 164-172.
148
John Strand
[lo] Y Ye and Sudhir Dixit, “On Joint ProtectionRestoration in IP-Centric DWDM-Based Optical Transport Networks,” ZEEE CommunicationsMagazine, vol. 38, no. 6, June 2000, pp. 174-183. [l 11 R. D. Doverspike, S. Phillips, and J. R. Westbrook, “Future Transport Network Architectures,” IEEE CommunicationsMagazine, vol. 37, no. 8, August 1999, pp. 96-101. [12] M. W. Maeda, “Management and Control of Transparent Optical Networks,” ZEEEJ. on SelectedAreas in Communications,vol. 16, no. 7, Sept. 1998, pp. 10081023. [13] B. Ramamurthy and B. Mukherjee, “Wavelength Conversion in WDM Networking,” IEEE J. on Selected Areas in Communications, vol. 16, no. 7, Sept. 1998, pp. 1061-1073. [14] H. Zang, J. P. Jue, and B. Mukherjee, “A review of routing and wavelength assignment approaches for wavelength-routed optical WDM networks,” Optical Networks Magazine ZEEE J. on Selected Amas in Communications, vol. 1, no. 1, Jan. 2000. [15] 0. Gerstel and R. Ramaswami, “Optical Layer Survivability-An Implementation Perspective,”IEEE J. on Selected Areas in Communications, vol. 18, no. 10, Oct. 2000, pp. 1885-1899. [ l a S. Baroni, J. 0. Eaves, M. Kumar, M. A. Qureshi, A. Rodriguez-Moral, and D. Sugerman, “Analysis and Design of Backbone Architecture Alternatives for IP Optical Networking,” IEEE J. on Selected Areas in Communications, vol. 18, no. 10, Oct. 2000, pp. 1980-1994. [17] D. Zhou and S. Subramaniam, “Survivability in Optical Networks,” ZEEE Network, vol. 14, no. 6, Nov. 2000, pp. 16-23. [18] T. H. Wu, Fiber NetworkSurvivability, Nonvood, MA: Artech House, 1992. [19] J. L. Strand, A. L. Chiu, and R. Tkach, “Issues For Routing In The Optical Layer,” ZEEE CommunicationsMagazine, vol. 39, no. 2, Feb. 2001, pp. 81-87. [20] C. W. Chao, P.M. Dollard, J. E. Weythman, L. T. Nguyen, and H. Eslambolchi, “FASTAR-A Robust System for Fast DS3 Restoration,” Proc. ZEEE GLOBECOM1991, Phoenix, Arizona, Dec. 1991, pp. 1396-1400. [21] A. Fumagalli and L. Valcarenghi, “IP Restorationvs. WDM Protection: Is There an Optimal Choice?’ ZEEE Network, vol. 14, no. 6, Nov. 2000, pp. 34-41. [22] 0. Gerstel, “Optical Layer Signaling: How Much Is Really Needed?’ ZEEE CommunicationsMagazine, vol. 38, no. 10, Oct. 2000, pp. 154-160. [23] S. Verma, H. Chaskar, and R. Ravikanth, “Optical Burst Switching: A Viable Solution for Terabit IP Backbone,” IEEE Network, vol. 14, no. 6, Nov. 2000, pp. 48-53. [24] G. M. Bernstein, J. Yates, and D. Saha, “IP-Centric Control and Management of Optical Transport Networks,” IEEE CommunicationsMagazine, vol. 38, no. 10, Oct. 2000, pp. 161-167. [25] A. Copley, “Optical Domain Service Interconnect: Defining Mechanisms for Enabling On-Demand High-speed Capacity from the Optical Domain,” ZEEE CommunicationsMagazine, vol. 38, no. 10, Oct. 2000, pp. 168-174.
3. Optical Network Architecture Evolution
149
[26] B. Rajagopalan, D. Pendarakis, D. Saha, R. S. h a m o o r t h y , and K. Bala, “IP Over Optical Networks: Architectural Aspects,” IEEE Communications Magazine, vol. 38, no. 9, Sept. 2000, pp. 94-103. [27] C. Qiao, “Labeled Optical Burst Switching For IP-Over-WDM Integration,” IEEE Communications Magazine, vol. 38, no. 9, Sept. 2000, pp. 104115. [28] D. K. Hunter and I. Andonovic, “Approaches to Optical Packet Switching,” IEEE Communications Magazine, vol. 38, no. 9, Sept. 2000, pp. 116-122. [29] M. Listanti and R. Sabella, “Architectural and Technological Issues for Future Optical Internet Networks,” IEEE Communications Magazine, vol. 38, no. 9, Sept. 2000, pp. 82-92. [30] R. Tkach, E. Goldstein, J. Nagel, and J. Strand, “Fundamental Limits of Optical Transparency,” Optical Fiber Communication ConJ, Feb. 1998, pp. 161-162.
[31] E. Modiano and A. Narula-Tam, “Mechanisms for Providing Optical Bypass in WDM-Based Networks,” OpticalNetworh Magazine, vol. 1, no. 1, Jan. 2000, pp. 11-18. [32] H. Zang, J. P. Jue, and B. Mukherjee, “A Review of Routing and Wavelength Assignment Approaches for Wavelength-Routed Optical WDM Networks,” Optical Networks Magazine, vol. 1, no. 1, Jan. 2000, pp. 4740. [33] C. P. Laresen and F! 0. Anderson, “Signal Quality Monitoring in Optical Networks,” Optical Networks Magazine, vol. 1, no. 4, Oct. 2000, pp. 17-23. [34] G. Hjilmtjrsson, P. Sebos, G. Smith, and J. Yates, “Simple IP Restoration for IP/GbE/lO GbE Optical Networks,” Optical Fiber Communication ConJ, 2000, vol. 4,2000, pp. 275-277. [35] S. Verma, H. Chaskar, and R. Ravikanth, “Optical Burst Switching: A Viable Solution for Terabit IP Backbone,” IEEE Network, vol. 14, no. 6, Nov. 2000, pp. 48-53. [36] J. Strand, “Optical Layer Services Framework,” Optical Interworking Forum oif2000.108, May, 2000; available from the author. [37] J. Strand and M. Lazer, “Some Routing Constraints,” Optical Interworking Forum oif 2000.109, May, 2000; available from the author. [38] M. Eiselt, “The Quest For Optimum Transmission Capacity: Is There An Optimum Fiber Choice,” OFC 2000 tutorial. [39] A. Odlyzko, “Internet Growth Myth and Reality, Use and Abuse,” iMP: Information Impacts Magazine, November 2000. [40] A. Odlyzko, “The Internet and Other Networks: Utilization Rates and Their Implications,” Feb. 26, 2000. Available at http://www.research.att.com/-amo/ doc/complete.html. [41] See http://www.canet2.net/library/papers.html. [42] B. Ramamurthy, H. Feng, D. Datta, J. P. Heritage, and B. Mukherjee, “Transparent vs. Opaque vs. Translucent Wavelength-routed Optical Networks,” Optical Fiber Communication Conference, 1999, and the International Conference on Integrated Optics and Optical Fiber Communication. OFC/IOOC ‘99.TechnicalDigest, vol. 1, 1999, pp. 5 M 1 .
150
John Strand
[43] OIF Carrier Group, Yong Xue, and Monica Lazer, eds, “Carrier Optical Services Framework and Associated Requirements for UNI,” Optical Interworking Forum oif2000.155, Sept. 13, 2000. Also available as working documents in TlX1.5 and the IETF. [44] G. Mohan and C. S. R. Murthy, “Lightpath Restoration in WDM Optical Networks,” IEEE Network, vol. 14, no. 6, Nov./Dec. 2000, pp. 24-33. [45] E. L. Goldstein, L. Y Lin, and R. W. Tkach, “MultiwavelengthOpaque OpticalCrossconnect Networks,” IEICE Duns. Commun., vol. E82-B, no. 8, Aug. 1999, pp. 1095-1104. [46] F. Bentivoglio and E. Iannone, “The Opaque Optical Network,” OpticaZ Networks, vol. 1, no. 4, Oct. 2000, pp. 24-31. [47] P. Green, “Progress in Optical Networking,” IEEE Communications Magazine, vol. 39, no. 1, Jan. 2001, pp. 54-61. [48] G. Miller, K. Thompson, and R. Wilder, “Wide-Area Internet Traffic Patterns and Characteristics,” IEEE Network, vol. 11, no. 6, Nov. 1997. [49] A. Banerjee, J. Drake, J. F! Lang, B. Turner, K. Kompela, and Y Rekhter, “Generalized Multiprotocol Label Switching: An Overview of Routing and Management Enhancement,” IEEE Communications Magazine, vol. 39, no. 1, Jan. 2001, pp. 144-150. [50] P. Arijs, B. van Caenegem, P. Demeester, F! LaGasse, W. Van Parys, and P. Achten, “Design of Ring- and Mesh-Based WDM Transport Networks,” Optical Networks, vol. 1, no. 3, July 2000, pp. 25-38. [51] R. H. Cardwell, 0. J. Wasem, and H. Kobrinski, “WDM Architectures and Economics in Metropolitan Areas,” Optical Networks, vol. 1, no. 3, pp. 41-50. [52] B. St. Amaud, “Overview of the Latest Developments in Optical Internets,” Optical Networks, vol. 1, no. 3, pp. 51-54. [53] C. Blaizot, I. Cerutti, A. Fumagalli, M. Tacca, L. Valcarenghi, R. Jagannathan, A. Lardies, F. Masetti, and M. Garnot, “Design and Planning of Transport Networks Based on SDH/SONET and WDM Optical Networks: Methods, Tools and Applications,” Optical Networks, vol. 1, no. 3, pp. 55-65. [54] Vinod Khosla, “Terabit Tsunami,” Keynote Address at OFC 2000, Baltimore, MD, March 2000. [55] K. G. Coffman and A. M. Odlyzko, “Internet Growth Is there a “Moore’s Law” for Data Traffic?” Preliminary version, 7/11/00. Available at: research.att.com/-amo. [56] L. Xu, H. Perros, and G. Rouska, “Techniques for Optical Packet Switching And Optical Burst Switching,”IEEE CommunicationsMagazine, vol. 39, no. 1, pp. 136-145. [57] A. Odlyzko, “Data Networks Are Mostly Emptly And For Good Reason,” ITPro, MarcWAprill999, pp. 6749. [58] B. St. Amaud, “Scaling Issues on Internet Networks” (draft), Canarie, Inc., Jan. 13,2001, available at www.canet2.nefflibraqdpapers.
3. Optical Network Architecture Evolution
151
[59] W. Norton, “Internet Service Providers and Peering:’ North American Network Operators Group (NANOG), 2000, available at www.nanog.org/mtg001Oltree.doc. [60] G. Gilder, “The Post-Diluvian Paradigm,” Gilder Technology Report, vol. V, no. 2, Feb. 2000. [61] “Fiberoptic Networks of Long-Distance Carriers in North America: Market Developments And Forecast,” KMI Corp., Newport, R.I., Nov. 1999. [62] S.Singh, The Code Book, New York: Anchor Books, 2000. [63] A. Saleh, L. Benmohamed, and J. Simmons, “Proposed Extensions to the UNI for Interfacing to a Configurable All-Optical Network,” Optical Interworking Forum contribution oif 2000.278, Nov. 7,2000. [64] L. Garrett, R. Vodhanel, S. Patel, R. Tkach, and A. Chraplyvy, “Dispersion Management in Optical Networks,” Proc. Eur. Cod. on Optical Comxnun., Edinburgh, Scotland, 1997. [65] ITU-T G.805, “GenericFunctional Architectureof Transport Networks,” 1995. [66] ITU-T G.803, “Architecture of Transport Networks Based on the Synchronous Digital Hierarchy,” 2000. [67] ITU-T G.872, “Architecture of the Optical Transport Network,” 1999. [68] J. Heiles, “Alignment of the UNI with ITU terminology:’ OIF Contribution oif2000.197, Sept. 1,2000. [69] ITU-T G.709, “Network Node Interface For The Optical Transport Network,” initial version, Feb. 2001. [70] A. Malis and W. Simpson, “PPP over SONETlSDH,” IETF RFC 2615, June 1999. [71] J. Kraushaar, “Fiber Deployment Update: End of Year 1998,” U.S. Federal Communications Commission, 1999. [72] R. Ballart and Y-C. Ching, “SONET: Now It’s the Standard Optical Network,” IEEE Communications Magazine, vol. 29, no. 3, 1989, pp. 8-15. [73] J. Berthold, “SONET and ATM,” in I. Kaminow and T. Koch, eds, OpticaI Fiber CommunicationsIIIA, Boston: Academic Press, 1997. [74] J. Yates, J. Lacey, and M. Rumsewicz, “Wavelength Converters in DynamicallyReconfigurableWDM Networks,” IEEE Communications Surveys, vol. 2, no. 2, 1999, www.comsoc.org/pubs/surveys. [75] M. Garey and D. Johnson, Computers And Intractibili-A Guide to the Theory of Np-Completeness, New York W. H. Freeman, 1979. [76] J. Strand, R. Doverspike, and G. Li, “Importance Of Wavelength Conversion In An Optical Network,” Optical Networks Magazine, vol. 2(3), MaylJune 2001, pp. 3344. [77l T.-H. Wu, “Emerging Technologiesfor Fiber Net Survivability,” IEEE Communications Magazine, vol. 35, no. 2, 1995, pp. 58-74. [78] ITU-T, G.841, “Types And Characteristics Of SDH Network Protection Architectures,” 1998. [79] ANSI, TI. 105.01, “SynchronousOptical Network (SONET)-Automatic Protection Switching,” 1999.
152
John Strand
[80] G. HjilmtJisson, J. Yates, S. Chaudhuri, and A. Greenberg, “Smart RoutersSimple Optics: an Architecture for the Optical Internet,” IEEE/OSA Journal of Lightwave Technology, Dec. 2000, pp. 37-50. [81] B. Doshi, S. Dravida, P. Harshavardhana, and M. A. Qureshi, “A Comparison of Next-Generation IP-Centric Transport Architectures,”Bell Labs Tech. J., vol. 3, no. 4,Oct.-Dec. 1998, pp. 137-143. [82] M. Reardon and S. Saunders, “Terabit Trouble,” Data Communications, Aug. 1999, pp. 11-16. [83] S. Chaudhuri and E. Goldstein, “On The Value of Optical-Layer Reconfigurability in IP-Over-WDM Lightwave Networks,” ZEEE Photonics Technology Letters, vol. 12, no. 8, Aug. 2000, pp. 1097-1099. [84] B. Gleeson, A. Lin, J. Heinanen, G. Armitage, and A. Malis, “A Framework for IP-Based Virtual Private Networks,” RFC 2764, IETF, Feb. 2000. [85] R. Doverspike, S. Phillips, and J. Westbrook, “Transport Network Architectures in an IP World,” Proc. INFOCOM-2000, March 2000, Tel-Aviv, Israel. [86] 0. Gerstel, “Optical Layer Signaling: How Much Is Really Needed,” ZEEE CommunicationsMagazine, vol. 38, no. 10, Oct. 2000, pp. 154-160. [87l D. Comer, Internetworking with TCP/IR El. I : Principles, Protocols, and Architecture, Upper Saddle River, NJ: Prentice Hall, 1995. [88] U. Black, ZP Routing Protocols: Ne OSPF:BGE P W I & Cisco Routing Protocols, Upper Saddle River, NJ: Prentice Hall, 2000. [89] J. Moy, OSPF:Anatomy OfAn Internet Routing Protocol, Boston: Addison Wesley Longman, 1998. [90] J. Stewart 111, BGP4: Inter-Domain Routing in the Internet, Boston: AddisonWesley Longman, 1998. [91] B. Davis and Y Rekhter, MPLS: Technology And Applications, San Francisco: Morgan Kaufmann, 2000. [92] D. Awduche et al., “Multiprotocol Lambda Switching:Combining MPLS Traffic Engineering with Optical Crossconnects,” Internet draft, draft-awduche-mplste-optical-02.txt, March 2000, work in progress. [93] L. McAdams and J. Yates, eds., “User to Network Interface Service Definition and Connection Attributes,” Optical Interworking Fonun Contribution oif2000.061, Dec. 15,2000. [94] S. Chaudhuri, G. HjPlmtJisson,and J. Yates, “Control of Lightpaths in an Optical Network,” work in progress, Internet draft, draft-chaudhuri-ip-olxc-controlOO.txt, 2000. [95] A. Chiu, J. Strand, R. Tkach, and J. Luciani, “Features and Requirements for the Optical Layer Control Plane,” work in progress, Internet draft, draft-chiustrand-unique-olcp.tt, Feb. 2001, available from the author also. [96] M. Blanchet, E Parent-Viagenie, and B. St-knaud, “Optical BGP (OBGP): InterAS lightpath provisioning,” work in progress, Internet draft, draft-parentobgp-OO.txt, Jan. 2001, available at www.canet2.netAibraxy/papers. [97] J. Strand and Y Xue, “Routing for Optical Networks With Multiple Routing Domains,” Optical Interworking Forum Contribution oif 2001.046, Jan. 2001; available from the authors.
3. Optical Network Architecture Evolution
153
[98] G. Bernstein and B. Rajagopalan, “Optical Inter-Domain Routing Requirements,” Internet draft, draft-bernstein-optical-bgp.txt,Feb. 2001, work in progress. [99] Bellcore, “SONET Bidirectional Line-Switched Ring Equipment: Generic Criteria,” GR-1230-CORE,Issue 2, Nov. 1995. [1001 Bellcore, “SynchronousOptical Network (SONET) Generic Criteria,” GR-253CORE, Issue 1, Dec. 1994. [loll Quoted by Russell McGuire, VP, Strategic Development Williams Corp., in a 12/3/1999 talk to the Risk Conference on Bandwidth Trading, New York. [lo21 A. Chiu and J. Strand, “Joint IP/Optical Layer Restoration After a Router Failure,” OFC2001, Anaheim, vol. 1, March 2001, pp. MN5-1-MN5-2. R. Doverspike and J. Yates, “Challenges for MPLS in Optical Network [lo31 Restoration,” IEEE Communications Magazine, vol. 39, no. 2, Feb. 2001, pp. 8%96. [lo41 R. Barry, “Optical Networking Technologies: Driving the Evolution of the Public Network,” NFOEC, 1999. [lo51 B. Mukherjee, “WDM Optical Communication Networks: Progress and Challenges,” IEEE Journal on Selected Areas In Communications, vol. 18, no. 10, Oct. 2000, pp. 181CL1824. 11061 S. Baroni, J. Eaves, M. Kumar, M. Qureshi, A. Rodriguez-Moral, and D. Sugrarman, “Analysis and Design of Backbone Architecture Alternatives for IP Optical Networking,” IEEE Journal on Selected Areas In Communications, vol. 18, no. 10, Oct. 2000, pp. 198CL1994. [lo71 E. Messmer, “Holding the Line on Call-Center Sprawl,” Network World, Apr. 2, 2001. [1081 R. Bhandari, Survivable Networks--Algorithm for Diverse Routing, New York: Kluwer Academic Publishers, 1999.
Chapter 4
Undersea Communication Systems
Neal S. Bergano ljco Telecommunications.Eatontown, New Jersey
4.1 Introduction In today’s Internet age information flows across continents as easily as it flows across the office. With so many “point and click” virtual connections, it is easy to forget that the world’s communicationsneeds are made possible by real systems based on fiber-optic cables. This comes as no surprise to those of us in the optics community; however, many others underestimate the importance of undersea fiber-optic cables for intercontinental telecommunications. The earth‘s continents are connected with a web of undersea fiber-optic cables that join the world’s major population centers. Anyone who makes international phone calls, sends international faxes, or simply surfs the Web at sites in other continents uses undersea fiber-optic cables. Although the debate regarding the merits of cable systems versus satellite systems for international communications ended many years ago, with cable systemsthe clear economicand technological winner, many people still assume that overseas communicationsoccur via satellite. But consider this: since 1990 over 600,000 km of undersea fiber-optic cable has been installed across the world’s oceans. Today, the vast majority of international telecommunications is carried on undersea cable^.^ This chapter reviews concepts for the design of long-haul transmission systems based on optical amplifierrepeaters and wavelength division multiplexing (WDM) techniques. Important strides have been made in areas of dispersion management, gain equalization, and modulation formats, which have made possible the demonstration of capacities exceeding 2TB/s on a single fiber. This chapter includes sections on the history of undersea cable, the amplified transmission line, dispersion and nonlinearity management, transmission formats, measures of system performance, error correcting codes, polarization effects, long-haul system design, experimental techniques, and future trends in long-haul optical transmission systems. ‘3’
4.2
A Rich History of Undersea Cables
Undersea cable has been in use for over 130 years. The installation of the first transatlantic telegraph cable was completed in 1858 (Fig. 4.1). Unfortunately, 154 OPTICAL FIBER TELECOMMUNICATIONS, VOLUME IVB
Copyright Q 2002, Elsevier Science (USA). All rights of reproduction in any form reserved. ISBX 0-12-395173-9
4. Undersea Communication Systems
155
Fig. 4.1 The crew of the 3200-ton HMS Agameemnon, which laid the first, briefly successful cable in 1858, watches anxiously, knowing that the mere wake of a passing whale might be forceful enough to break the cable. (The Atlantic Cable, Burndy Library, Norwalk, CT, 1959.)
this cable only worked for a few weeks before a problem with the cable rendered it unusable. The first successful transatlantic telegraph cable connected North America to Europe and went into service in 1866,34 years after Samuel Morse invented the telegraph. At that time an experienced telegraph operator could send about 17 words per minute, at a cost of about $5 per word.4 It took nearly 90 years from the time of the first telegraph cable to install the first transatlantic telephone cable. In 1956, the first TAT (Trans Atlantic Telephone) cable went into service, providing 48 telephone circuits between Newfoundland and Scotland.’ These analog systems were based on coaxial cables with electronic amplifiers. They eventually grew in capacity to over 4200 voice circuits for systems installed as recently as 1983. During this time, the capacity of transatlantic cable’s circuits increased at an annual rate of -20% (Fig. 4.2). These circuits were transmitted by frequency division multiplexing many circuits over an electrical bandwidth of a few tens of megahertz. The signals were boosted in “wide-band’’ electrical amplifiers that were placed in repeaters and spaced every 9.5 km. In an interesting twist of fate, the cable in the coaxial systems was linear, while great design efforts were expended to cope with the nonlinearity of the electronic amplifiers, which is opposite from the present optically amplifier fiber-optic systems. The first undersea fiber-optic systems were installed across the Atlantic and Pacific Oceans in 1988-1989 and had a capacity of 280 Megabithec on each
156
Neal S. Bergano TWO-WAY CABLE CIRCUITS 64Kbk Digital Circuits 3KHz Analoa Circuits
> I 00% Annual Growth
1M 100K-
10K1K-
100 -
1960 1955
1970 1965
1980 1975
1990 1985
2000
1995
2005
YEAR
Fig. 4.2 The cumulative capacity that has been installed crossing the Atlantic Ocean over the past 4 decades. The capacity is given in 3-kHz circuits for the analog systems, and 64-kB/s circuits for the digital systems.
of three fiber pairs.6 These systems were actually hybrid optical systems in the sense that the repeaters converted the incoming signals from optical to electrical, regenerated the data with high-speed integrated circuits, and retransmitted the data with a local semiconductor laser. The transmission capacity of the regenerated fiber cables eventually increased to 2.5 Gigabidsec, and repeater spacing increased with the switch from 1.3 pm multifrequency lasers to 1.55 pm single-frequencylaser diodes. These first undersea fiber systems revolutionized international telephony, bringing costs down and greatly expanding capacity and quality. However, the ability of the regenerator systems to exploit the large fiber bandwidth was limited by the capacity bottleneck in the high~~ speed electronics of undersea repeaters. Beginning in the mid- 1 9 9 0 undersea fiber-optic systems using erbium-doped fiber-amplifiersin the repeaters were deployed. These systems removed the electronic bottleneck and provided the first clear optical channel connectivity between the world’s continent^.^ Today’s undersea systems use erbium-doped fiber amplifiers (EDFAs) to compensate the attenuation in the optical fiber cable. These optical amplifiers are in repeaters that are typically spaced every 50 km along the cable and have an optical bandwidth that is wide enough to support many optical channels using WDM techniques. Data signals coming from the land-based systems (typically referred to as “terrestrialyysystems) are converted to an optical format that is more robust for transmission over transoceanic distances. Systems being designed today for deployment in the next few years will support many
158
Neal S. Bergano
,*\
STRENGTH WIRES
OPTICAL FIBER
Fig. 4.4 Cut away diagram of a cable section.
proliferation of lightwave cable systems. The EDFA is a nearly ideal building block for providing optical gain in a lightwave communications systems.8 EDFAs can be made with a variety of gains in both the “Conventional” wavelength band (C-band) from 1526-1566 nm and the “Long” wavelength band (L-band) from 1566-1606 nm.* Because the EDFA is a fiber device, it can be easily connected to telecommunication fiber with low loss and low polarization dependence. Most importantly, EDFAs can be manufactured with the 25-year reliability that is required for use in undersea systems. From an optical standpoint, the undersea portion of the system (Fig. 4.5) is sometimes referred to as an “amplifier chain”; it is a concatenation of optical amplifiers and cable spans. The attenuation in optical fiber is 0.2 dB/km (at 1550nm); thus, in a 6000-km-long transatlantic system, the optical amplifiers will compensate a total of 1200dB of cable attenuation! For proper system operation where there is a tight tolerance on the power launched into the transmission fiber, the gain in the amplifiers must exactly match the attenuation in the fiber. At first this might seem like a difficult task; however, the solution comes almost for free from the gain characteristics of the optical amplifier. One of the many advantages of the EDFAs is the ability to be operated deep into gain compression without any significant distortion of the high-speed data signals being transmitted. A natural “automatic gain control” occurs by * These are approximate values, since the wavelength range ofthe C- and L-bands are somewhat arbitrary.
4. Undersea Communication Systems
159
Fig. 4.5 A typical undersea cable link. The undersea equipment consists of cable sectionsjoined by repeaters, which house the EDFAs. The terminal equipment grooms the “terrestrial-grade” signals for transmission across the ocean.
t
Operating Point
Gain (Span Loss)-’ Output Power
Fig. 4.6 The amplifier’s output power is controlled by operating each EDFA with gain compression. Steady-state operation occurs where gain equals inverse attenuation in the spans.
designing the small signal gain of the amplifier to be larger than the attenuation in the cable section (Fig. 4.6). Thus, if for some reason the power should drop in any particular section of cable (Fig. 4.7), the following amplifier will see a smaller input power, resulting in an increase in gain so as to establish proper operating power in the rest of the optical path. (Note: Strictly speaking, this simple automatic gain control (AGC) argument holds only for narrowband operation. For wide-band amplifier chains, any mismatch between the amplifier’s gain and the span’s attenuation results in gain tilt.)
160
Neal S. Bergano
r I, 71 ti f r r , -
Repeater output Power
r
Normal Output Power
_ _ _ _ ____----------- --- --- -
_ ____ _ _ _ _ _ _ _ ___ ---- ---
--- -
Position of Repeater in System
Fig. 4.7 Repeater output powers versus transmission distance. The repeater’s output power is controlled by gain compression in the amplifiers.
The real challenge is to equalize the power over a wide optical bandwidth to allow for many WDM channels. Here we are not as fortunate as with the total power control given by the EDFA gain compression. Most of the gain equalization is accomplished using passive gain-flattening filters placed along the length of the amplifier chain.9These filters can be packaged with the individual amplifiers to provide the majority of the gain equalization. Depending on the required level of equalization, additional “clean-up” filters can be placed at regular intervals (i.e., once every 10-20 amplifiers) along the amplifier chain. Figure 4.8 shows the gain shape of a single-stage EDFA before and after gain equalization. This amplifier was designed to have usable gain throughout the entire C-band. The amplified spontaneous emission (ASE) noise accumulation can be controlled by properly selecting the distance between amplifiers. In a long transmission line, ASE noise generated in the EDFAs can accumulateto power levels similar to the data-carrying signal. The accumulated noise can influence the system’s performance by reducing the level of the signal and signal-tonoise ratio. The noise power out of an optical amplifier is proportional to the amplifier’s gain and is given by:’O
where hv is photon energy, g is gain in linear units, nsp is the excess noise factor related to the amplifier’s noise figure, and Bo is optical bandwidth. For a chain of identical amplifier spans, the total accumulated ASE noise out of the N* amplifier is simply NPnoise(assuming no signal decay caused by noise accumulation). As a consequence, the spectral density of the accumulated noise at the end of the system depends on the repeater gain and fiber loss. Consider a linear system where we treat only noise accumulation, with a fiber loss of 0.2 dB/km.
4. Undersea Communication Systems Erbium Doped Fiber
Gain Flattening Filter
xmxwx
Wavelength Selective
1
*
Signal out
V
*O
161
Isolator
Tap Coupler
Coup’er
1525 1530 1535 1540
1545
1550 1555 1560 1565 1570
Wavelength (nm)
Fig. 4.8 Measured gain vs wavelength for a full C-band EDFA. The gain is measured with and without the gain-flatteningfilter for a “flat” input power.
A 150-km repeater spacing would require 30-dB amplifiers, whereas a 50-km spacing needs three times as many 10-dB amplifiers. A 30-dB amplifier (