Cisco - Frontier Global Center

Frontier GlobalCenter Alan Hannan Director, Network Architecture & Engineering July 14th and 15 th, 1999

Overview n Using Fiber to build an IP Network n Building a Scalable Network Architecture n Building a Scalable POP Architecture n BREAK n IP/DWDM and POS Issues n MPLS And Traffic Engineering n Q&A and Discussion

Slide 2

Cisco Networkers

1

Overview n Using Fiber to build an IP Network n Building a Scalable Network Architecture n Building a Scalable POP Architecture n BREAK n IP/DWDM and POS Issues n MPLS And Traffic Engineering n Q&A and Discussion

Slide 3

Cisco Networkers

Using Fiber to Build an IP Network n Primary Physical Asset is Fiber 9

k

n Secondary Physical Asset is Optronics 7

m

Optical/Electronic Infrastructure to “power” fiber Seattle

Heelna Spokane

.

2 5

8

593.3 k1m

31m k.722

.28 25 . 11 77

4

5

886.4km 68.46 m k

.

.4km 4069

St.Louis

mk

dlei f tsW e

2.011 4

21851.km

k

km 2451.4 Atlanta

m 1601.9k Greenvile

Birmingham

4

3

.6km 6454

Greensboro 1287.5 7 RockyMount km 109m k8. CharlotteRaleigh

Chattanooga

12

3 m

2

94m k.581

Nashville 2492.4k m

m Tulsa 1882.9k OklahomaCity

Albuquerque

Phoenix

01.53 Fort Worth k1m Dallas

Macon

.

San Deigo

Anaheim

k.191 39m

9 9m 1k

m

1873.1k m 3 23 . Los Angeles 4 km

k.02 05m

.

k

1

18590.km

Santa Barbara

Richmond 1676.8k m Portsmouth m 1920.3k

2

4 2

San Luis Obispo

6

39m k.901

BowlingGreen 0

Salinas

9

36 m . k324

0 Las Vegas

7

Fredricksburg

Louisville 92m k.146

Oakland

San Jose

1k m 4 49. 7 mk268.

14287.km

2

9

2 2

1054 .1km Kansas Ctyi 4695 .1km Topeka

86172m k.

9

San Francisco

k m

k

2 0

16.8 . 1m k

3

7

0 2 1

8971 .3km

Colorado Springs

1 . 1 0m k 5 4

0 2 1k 48mk.212 . 52 2 2. m 9 .09mk 21

13

13.801mk

Lincoln

Denver

9098 .4km

.0km 9316 Reno

m 2147.2k

Rochester (Fitzhugh) (Syracuse SyracusePOP) flaPloOPJn. 2179.1k .4 (Buf Bua fo ) 96.88km 1337 m Ab km l any1 Battle 93.41km (Detroit 915.1km Creek Southe fid l) 97 0.6 .8km Boston km Poughke 1700 034.2km Francisco (Poughkeeeppssie Jn W2aterford eiPOP) . Providence Cleveland 1 1979.5km (Erie) 49.o7o3dkm 68km .24 k (Monroe) W ry 1282.7 m .9km 512.6 8.1km New Toe l do 1739 62.Pl 7b6ukm 84 km White3 New London Youngstown 83 .kmains 241 StamfordHaven Newark .km .4km 2106 1823 21km .73NewYork .6km 1790 .7km Trenton Pittsburgh Altoona Harrisburg Philadelphia .3km Cou Indianapolis 1147 l mbus m 1763.0km Dayton 1569.1k Baltimore Cincinnati Washn i gton

South Bend

24.88k m

18 1 1 k k4 6. m 08 m

k m

.

.

Redding

k.443 16m

8

6

8

8

Chicago 1382.5km

.9km 2221 Des Mon ies

Omaha

117.5km

1m k

8 3

k 2

Salt Lake City

Chico

Davis (Sacramento)

GreenBay

Mw li aukee

Medford

8978.km

k

1391 lrie 3 .8km Eau Ca 065 .3km

Owatoona

.7km 3 . 20751

k

m

m

1

9 1 9 m 7 1 3 . 39 . k 7 m

Minneapolis

Casper

9

4

. 4

8

Boise

Eugene

m

3

9 Billings

Portland

4

5030 .6km

8

El Paso

Jackson

34 74m.k1

Tucson

Mobile

9659.9k m 3085 .3km

67m k.931

Austin

Baton Rouge

Pensacola

Houston

10 Talahassee Jacksonvile

NewOrleans

San Antonio

DaytonaBeach Orlando Tampa

11 Ft.Myers

W.PalmBeach Ft.Lauderdale Ma i mi

Slide 4

Cisco Networkers

2

Using Fiber to Build an IP Network n Optronics Network Overview

WDM SONET ADM

ATM

WDM IP

MUX ADMs (1:3, etc..)

SONET ADM

ATM

IP

MUX ADMs (1:3, etc..)

Slide 5

Cisco Networkers

Using Fiber to Build an IP Network n Fiber between cities uses Optical

Amplifiers and Regenerators n Optical Amplifiers • “OA” • Take signal, and using Laser Pump, increase magnitude n Regenerators • “Regens” • Take signal, re-clock and re-synchronize Slide 6

Cisco Networkers

3

Overview n Using Fiber to build an IP Network n Building a Scalable Network Architecture n Building a Scalable POP Architecture n BREAK n IP/DWDM and POS Issues n MPLS And Traffic Engineering n Q&A and Discussion

Slide 7

Cisco Networkers

Building a Scalable Network Architecture

n What does “scalable” mean? • Able to grow with minimal change • Growth is repetition of simple concepts, not new design n How is FGC Scalable? • Network is composed of many ‘sub-networks’ • Each sub-network is a module, which can be placed anywhere

Slide 8

Cisco Networkers

4

Building a Scalable Network Architecture

n Sub-Networks • Each “Sub-Network” is a ring —Redundancy —Resiliency

• WAN Ring(s) —Wide Area Network

• LAN Ring(s) —Local Area Network

• MAN Rings(s) —Metropolitan Area Networks Slide 9

Cisco Networkers

Building a Scalable Network Architecture

n Sub-Networks

Slide 10

Cisco Networkers

5

Building a Scalable Network Architecture

n Sub-Networks • Can Scale Upwards —Continents —Theatres —Planets —Etc…

• Can Scale Downwards —Cities —Neighborhoods —Buildings Slide 11

Cisco Networkers

Overview n Using Fiber to build an IP Network n Building a Scalable Network Architecture n Building a Scalable POP Architecture n BREAK n IP/DWDM and POS Issues n MPLS And Traffic Engineering n Q&A and Discussion

Slide 12

Cisco Networkers

6

Building a Scalable POP Architecture n POP Architecture • POP – Point of Presence • We call them “HUBs” —A topological collection of routers and switches that functions as one node on the greater network

• All Backbone Routers and Switches are redundant • No Single Point of Failure • Customer can arrange for redundant links to FGC for their own redundancy Slide 13

Cisco Networkers

Building a Scalable POP Architecture n POP Architecture FGC W.H.I.P. IP Hub

WR1

(v1.4, 4/16/99)

WR2

O C -4 8 O C -1 2 O C -3 o r l o w e r

AR1

CR1

BR1

CR2

HR1

DR1 Modems

ADMs

Slide 14

IXCs * LECs/CAP FRO

Ethernet Switches Cisco Networkers

7

Overview n Using Fiber to build an IP Network n Building a Scalable Network Architecture n Building a Scalable POP Architecture n BREAK n IP/DWDM and POS Issues n MPLS And Traffic Engineering n Q&A and Discussion

Slide 15

Cisco Networkers

BREAK

Break Slide 16

Cisco Networkers

8

Overview n Using Fiber to build an IP Network n Building a Scalable Network Architecture n Building a Scalable POP Architecture n BREAK n IP/DWDM and POS Issues n MPLS And Traffic Engineering n Q&A and Discussion

Slide 17

Cisco Networkers

IP/DWDM and POS Issues n Frontier uses 2 DWDM Systems • NEC —OC-48 —16 Lambda —40 Gbps System (714,000 DS0 Channels)

• Pirelli —OC-48 or OC-192 —128 Lambdas • 4 Chassis • Each Chassis does 32 Lambdas

—1.28 Tbps System (23,000,000 DS0 Channels) Slide 18

Cisco Networkers

9

IP/DWDM and POS Issues n We plug OC-48c Lambdas from both

systems directly into our Cisco 12xxx’s

Slide 19

Cisco Networkers

IP/DWDM and POS Issues n Routers and DWDM interoperated

surprisingly well n We use Single-Mode Short-Reach at all locations, DWDM does the long-range signal generation n OC-12c, OC-3c are done w/ DWDM SONET ADM interconnection n OC-48c and OC-192c are ‘thin-SONET’ with no APS Slide 20

Cisco Networkers

10

Overview n Using Fiber to build an IP Network n Building a Scalable Network Architecture n Building a Scalable POP Architecture n BREAK n IP/DWDM and POS Issues n MPLS And Traffic Engineering n Q&A and Discussion

Slide 21

Cisco Networkers

Traffic Engineering n Problem: Optimally fitting traffic to existing network

capacity and topology n Consisting of: • ~250 Routers, ~175 of which run IS-IS and full BGP • ~1000 Backbone links • ~20 “major” hubs, ~100 total hubs —3 areas [bay, dc, nyc] make up 80% of traffic • function of population density

• ~2.5 Gbps of Egress or Off-Net Traffic [aka Peering] —~16 Public Interconnection Interfaces —~80 Direct Interconnection Interfaces

• Backbone mostly OC-48c, Metros at N*DS3s, OC-3, OC-12

Slide 22

Cisco Networkers

11

Traffic Engineering Production

SEA1

Ready Current Working Circuit Pending Circuit

DS3 OC-3c OC-12c OC-48c

IP Network 5/20/98alan

Hub Legend

Circuit Legend

< DS3

MIN1

SEA

WHIP HUB

nonWHIP HUB

MSP

~T1 Hub

H u bw / A T M

M D C HUB

pending completion

MON

DET1

ROC1

BOS1

BOS

ORD QDV CHI1 DEN

BOI

ASH

NYC2 NYC2

DEN1

LGA2

DVX

LGA EWR

COS JFK SFO1

CLE1

PHI1 PSK

MCI

PAO N U Q 2

KCY1 S N V 2

CMH

NUQ

PHL

BWI

CGF

SNV SJC DAL

DAL1

ABQ

IAD

BUR

USC

ATL1

HOU

LAS

WDC1

LAX1

ATL

DCA

HOU1

RAL1 SAN

TPA

This Network Diagram contains:

TAM1 ORL1

ExistingCircuits

SAN1 PHX PHX1

ELP

A T MH u b s

AUS1

MFW

MIA1

NonWh p i Hubs Legend W H P IH u b s

TUS InterimMetroCircuits

Slide 23

LegacyHubTie-Inckts. NewCircuits-October NewCircuits-November NewCircuits-December NewCircuits-January NewCircuits-February NewCircuits-March NewCircuits-NoDate PendingMetroCircuits ProposedforNovember

Cisco Networkers

Network Topology (ideal) and Traffic Engineering A-End

B-End

Slide 24

Cisco Networkers

12

Network Topology (conceptual)

Traffic Engineering - MPLS Transit Backbone

Metro

Metro

Metro

Metro

Metro

Metro

Metro

Metro

Metro

Slide 25

Cisco Networkers

Traffic Engineering - MPLS n Must embrace Hierarchy to build scalable architecture n Target of 300 Routers in 9 regions n Designate Routers as • Regional • Backbone

n 9 regions, 1 backbone region, 30 routers/region n 30^2 == 9000 LSPs / Region n 9000 * 10 = 90,000 LSPs / System n Scalable both up and down (continents, neighborhoods)

Slide 26

Cisco Networkers

13

Traffic Engineering Framework

Traffic Engineering - MPLS

Slide 27

Cisco Networkers

Traffic Engineering - MPLS Benefits of Traffic Engineering with MPLS • Fine Grained Control over router-pair traffic flows —previous IGP manipulations complex

• Fine Grained statistics —Traffic levels on LSP Traffic between end points

• L3 Hop Masking —Obviate Win95 TTL Issue

• Provides Framework for Many Services —VPDN, VOIP, FR/IP, QOS, etc...

Slide 28

Cisco Networkers

14

Traffic Engineering - MPLS n MPLS Testing in Lab ~105 Days ago n Code Deployment in network ~75 Days ago n Production Implementation ~51 Days ago n Today: • Code Running on ~145 Routers • WAN CR Mesh between 7 Hubs • 10 Cities Meshed within Hub • NYC Region Mesh ~50% complete • Busiest LSPs are in regions near peering routers, ~100Mbps • Surprisingly, City_Router-Pair traffic flows de-aggregate significantly

n ETA for completion is 8/1/99

Slide 29

Cisco Networkers

Overview n Using Fiber to build an IP Network n Building a Scalable Network Architecture n Building a Scalable POP Architecture n BREAK n IP/DWDM and POS Issues n MPLS And Traffic Engineering n Q&A and Discussion

Slide 30

Cisco Networkers

15