502 0943_05F9_c2
1
© 1999, Cisco Systems, Inc.
Introduction to QoS in Campus Networks Session 502
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
2
1
Agenda • Networking Trends • QoS Demystified • QoS in Campus: Justifications • QoS Model and Techniques • Implementing QoS • Summary • Q&A 502 0943_05F9_c2
3
© 1999, Cisco Systems, Inc.
Networking Trends Evolution
502 0943_05F9_c2
Bridges to First Generation Routers
Early 1980’s
Multiprotocol Router, Dynamic Routing Protocols
Mid 1980’s
Multicast, Switching, Scaling, DHCP
1990’s
Single Network for Delay Sensitive (Voice), High Bandwidth (Video), Standard Data (E-mail) and MissionCritical Data (ERP)
Mid 1990’s–2000
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
4
2
Voice, Data, Video Communications You Have Mail
Ring ng i R
[email protected]
Voice
502 0943_05F9_c2
E-Mail
Internal Web
Video
© 1999, Cisco Systems, Inc.
Mission Critical
5
What Is Mission Critical?
• ERP applications • SNA applications • Selected physical ports • Selected hosts/clients
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
6
3
Enterprise Resource Planning (ERP) Networked Applications • Order entry • Finance • Manufacturing • Human Resources • Supply chain management • Sales force automation 502 0943_05F9_c2
7
© 1999, Cisco Systems, Inc.
Trend—Integrated Multiservice Network: Data, Voice, Video Data Voice
Video Internet
Single Infrastructure
Packet-Based Multiservice Network
• Integration of data, voice, and video services into a single packet-based infrastructure using IP • Both in enterprise and public service provider networks infrastructure 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
8
4
Data, Voice, Video Benefits of Integration Short Term
Benefits
Medium Term
Business/ Application Enablement
Long Term
Infrastructure/ Information Convergence
Budget Leverage 1998 502 0943_05F9_c2
2000
2002 9
© 1999, Cisco Systems, Inc.
Agenda • Networking Trends • QoS Demystified • QoS in Campus: Justifications • QoS Model and Techniques • Implementing QoS • Summary • Q&A 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
10
5
QoS—What Is It ? • A traditional network is best-effort • All traffic get the same service, i.e., the forwarding behavior by a network device is FIFO • QoS prioritizes traffic into different service levels and provides preferential forwarding treatment to some data traffic at the expense of lower priority traffic • QoS = Preferential treatment 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
11
QoS—Why Is It Needed? Motivation for QoS Is Manifold • Integrated networks carry different traffic types from a variety of business-enabling applications • Business drivers and policies dictate preferential treatment for some type of traffic over other(s) • Convergence of voice and data networks force us to consider servicing two different types of traffic on a single wire 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
12
6
Different Requirements Data
Voice
• Bursty
• Smooth
• Greedy
• Benign
• Loss Sensitive
• Loss Insensitive
• Delay Insensitive
• Delay Sensitive
• TCP Retransmits
• UDP Best Effort
502 0943_05F9_c2
13
© 1999, Cisco Systems, Inc.
QoS Components
Bandwidth Loss Delay Delay Variation (Jitter)
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
14
7
Not All Traffic Is Created Equal Data (Best-Effort)
MissionCritical Data
Voice
Video
Low to Moderate
Moderate to High
Moderate to High
Low to Moderate
Random Drop Sensitivity
Low
Low
High
Moderate to High
Delay Sensitivity
High
High
Low
Moderate to High
Jitter Sensitivity
High
High
Low
Low to Moderate
Bandwidth
502 0943_05F9_c2
15
© 1999, Cisco Systems, Inc.
What Is Driving the Need for Preferential Treatment in Networks ? Mission-Critical Apps Voice Video None Other
0
20
40
60
80
100
Source: Forrester, August ’98 Fortune 1000
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
16
8
Example: Evolving Campus Network Existing Network
New Additions
Voice
Video Surveillance
PCs
Mission Critical
MissionCritical Servers
E-mail Servers
Protect Mission-Critical Traffic! 502 0943_05F9_c2
17
© 1999, Cisco Systems, Inc.
Consolidated Network Distribution
Wiring Closet
Voice
PCs 502 0943_05F9_c2
Server Farm
MissionCritical Application
Video Surveillance
MissionCritical Application
Web Servers
Four Different Traffic Types PCs, Voice, Video, Mission Critical
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
18
9
Agenda • Networking Trends • QoS Demystified • QoS in Campus: Justifications • QoS Model and Techniques • Implementing QoS • Summary • Q&A 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
19
QoS in Campus Networks?
“
502 0943_05F9_c2
Why Not Just Increase the Link Speed? Throw More Bandwidth at the Problem! © 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
” 20
10
Maybe You Can, Maybe You Can’t • QoS in the campus is primarily about buffer management • Buffer management is required to control delay or drops • TCP will eventually retransmit dropped packets • Do you care about delay or drops? 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
21
Buffers Fill for Various Reasons
• Rate mismatch (1000M to 10M Ethernet) • Many to one (multiple interfaces talking to the same interface all with the same rate) • Aggregation points 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
22
11
TCP Traffic Behavior
Media Sender
Receiver Ack 21 1Ack’s Ack 4 Ack’s
Buffers are available at Sender and Receiver. Media is a buffer of sorts as well because of RTT Because of Almost Zero RTT in LAN, Switch Has to Buffer the TCP Window 502 0943_05F9_c2
23
© 1999, Cisco Systems, Inc.
TCP Traffic Behavior Example: Transmission of a 13 Segment Window Round Trip Time (RTT) of 100ms
6
502 0943_05F9_c2
5
4
3
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
2
1
24
12
TCP Traffic Behavior Window Perfectly Matched with RTT Example: Link Holds 12 IP Segments Worth of Data Which Is Equal to TCP Window 13
12
11
10
9
8
6
5
4
3
2
1
Ack
Next Window Is Transmitted upon the Receipt of Segment 1 Acknowledgement 502 0943_05F9_c2
25
© 1999, Cisco Systems, Inc.
TCP Traffic Behavior Window Not Perfectly Matched
9
8
7
6
1
2
3
4
5
Ack
Link Buffers, Less Than the TCP Window Which Causes Transmit Buffer to Fill 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
26
13
Buffer Congestion Speed Mismatch Most Traffic Is Server to Client Smaller Percentage of Traffic Is Client to Server 10 Meg
1000 Meg
Speed Mismatch
Speed Mismatch: Most Likely Point of Congestion on Output Ports of 10 Meg Interface
10 Mbps
1000 Mbps
502 0943_05F9_c2
27
© 1999, Cisco Systems, Inc.
Buffer Congestion Many to One
Switching Fabric
• Required when multiple ports are contending for the same port 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
28
14
Buffer Congestion Aggregation Point • Links oversubscribed
Aggregation
• Buffers full (clue: packets dropped) • Buffering reduces loss but delaysensitive application could be negatively impacted 502 0943_05F9_c2
29
© 1999, Cisco Systems, Inc.
Result Packets that Made through; Rest Are Dropped
Even though the Average Link Utilization Is Below 100%, Buffers Can Still Fill up
Buffers Link Utilization 60%
Example: 100 Mbps Link
Packets from Different Applications
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
30
15
What Happens When Multiple Packets Are Dropped Within the Same Window?
• Window size is reduced exponentially since it is halved for each failed ACK • TCP window may be reduced to the minimum • Transmitter of the sender tends to lock-up 502 0943_05F9_c2
31
© 1999, Cisco Systems, Inc.
Voice Delay Budget: 150 ms Cumulative Transmission Path Delay CB Zone Satellite Quality High Quality 0
100
Fax Relay, Broadcast 200
300
400
500
600
700
800
Time (msec) Delay Target
ITU G.114 Recommendation <= 150 msec 1-Way Delay
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
32
16
Buffer Management Delay in Campus Network
IP
IP
150ms
502 0943_05F9_c2
33
© 1999, Cisco Systems, Inc.
Conclusions • Buffers can congest in LANs • QoS required when there is congestion in buffers • Buffer Management can help reduce loss • Buffering reduces loss but delay sensitive application could be negatively impacted 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
34
17
Agenda • Networking Trends • QoS Demystified • QoS in Campus: Justifications • QoS Model and Techniques • Implementing QoS • Summary • Q&A 502 0943_05F9_c2
35
© 1999, Cisco Systems, Inc.
QoS Operational Model Admission Control
Allow Allow or or Dis-allow Dis-allow Frames/Packets Frames/Packets Based Based on on Application Application Type Type or or Policy Policy (Security, (Security, Time Time of of Day, Day, Etc.) Etc.) E.g., E.g., Application Application Recognition Recognition
Input Classification
Assigning Assigning aa Label Label Based Based on on Examining Examining Fields Fields in in the the L2 L2 (802.1p, (802.1p, ISL) Diffserv) ISL) or or L3 L3 Header Header (IP (IP Precedence, Precedence, DSCP DSCP Codepoint Codepoint from from Diffserv Diffserv))
Input Q Scheduling
Assigning Assigning the the Frame/Packet Frame/Packet to to an an Input Input Queue Queue and/or and/or Drop Drop Threshold Threshold Based Based on on the the Label Label Assigned Assigned
Policing Re-classification Output Q Scheduling 502 0943_05F9_c2
Ensuring Ensuring Traffic Traffic Rate Rate Is Is Within Within Specified Specified Contract Contract if if Policing Policing Is Is Turned Turned on. on. Excess Excess Is Is Dropped Dropped or or Marked Marked Down Down to to aa Lower Lower Priority Priority Assign Assign aa New New Label Label to to the the Frame/Packet Frame/Packet if if Necessary, Necessary, Based Based on on Configuration Configuration (Policy, (Policy, Security, Security, Congestion, Congestion, Etc.) Etc.) or or Policing Policing Action Action Assigning Assigning the the Frame/Packet Frame/Packet to to an an Output Output Queue Queue and/or and/or Drop Drop Threshold Threshold Based Based on on Original Original or or New New Label Label Assigned Assigned
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
36
18
Admission Control PeopleSoft HR Client
Oracle Database PeopleSoft
Oracle Print
ERP Clients SQL Listener ERP Server Farm WEB Traffic Port 80
WEB Server
WEB WEB NMS Games (Port 80) (Port 80) 502 0943_05F9_c2
37
© 1999, Cisco Systems, Inc.
Classify at Layer 3 and Layer 2 Standard IPV4: Three MSB Called IP Precedence (DiffServ May Use Six D.S. Bits Plus Two for Flow Control)
Layer 3 IPV4
Version ToS Length 1 Byte Len ID
Three Bits Used for CoS (Class of Service)
Layer 2 ISL ISL Header 26 Bytes
Offset TTL Proto FCS IP-SA IP-DA Data
Encapsulated Frame 1…24.5 KBytes
Three Bits Used for CoS (User Priority)
Layer 2 802.1Q/p PREAM. SFD 502 0943_05F9_c2
FCS 4 Bytes
DA
SA
TAG 4 Bytes
PT
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
DATA
FCS 38
19
Input Classification
• Priority can be based on ToS in a packet • Priority could also be determined by the 3bit user-priority field in the ISL/.1Q header This field is commonly called Class of Service (CoS)
• Value of 0 is for best effort and 7 is for the highest priority traffic 502 0943_05F9_c2
39
© 1999, Cisco Systems, Inc.
CoS Assignment
• Untagged packets, i.e., packets without an ISL/.1Q tag such as those on an access port are assigned an initial label by the ingress port • Tagged packets can carry a CoS assigned by say an end-station 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
40
20
Trusted vs. Untrusted • Cisco’s QoS model assumes that the CoS carried in a packet may or may not be trusted by the network device • For example, some servers maybe reliable enough to correctly tag the packets • End stations like user PCs can mostly not be trusted to tag a packet’s priority correctly 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
41
CoS for Untrusted Ports • “Incorrect Tag” means tag in a way different from what the administrative policy specifies • Network Administrators can designate which ports can be “trusted” to tag packets correctly • On untrusted ports, the ingress port assigns a default CoS to all packets— tagged or not 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
42
21
CoS Assignment (Cont.)
• On trusted ports, ingress port assigns CoS only to untagged packets • CoS assigned by an ingress port is independent of the VLAN, i.e., on an untrusted trunking port, packets of all VLANs get tagged by the same CoS 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
43
Random Early Detection (RED) • RED reduces long term average queue • Packet drops are randomized throughout queue depth
Transmit Buffer Queue
• Drop rate is increased as queue depth is increased 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
44
22
Random Early Detection Benefits
• Average delay reduced • Reduced TCP slow start conditions • Reduced global synchronization • Reduces negative bias towards light users 502 0943_05F9_c2
45
© 1999, Cisco Systems, Inc.
Weighted RED • WRED addresses: In the event packets need to be dropped, what class of packets should be dropped Packets Classified as Blue Start Dropping at a 50% Queue Depth. Drop Rate Is Increased as Queue Depth Is Increased
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
Queue
Packets Classified as Gold Are Dropped at 90% Queue Depth
46
23
RED for Congestion Avoidance Uncontrolled Congestion
Managed Congestion
• Maximize goodput • Accommodate burstiness • Minimize delay 100%
Offered Load Over Time 502 0943_05F9_c2
47
© 1999, Cisco Systems, Inc.
WRED Congestion Avoidance— Two Drop Thresholds
Standard Service Number of Packets Discarded
Adjustable “Slope”
Queue Length
502 0943_05F9_c2
Premium Service
Standard Minimum Threshold 1
Premium Standard Minimum Maximum Threshold 2 Premium Maximum
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
48
24
Weighted Random Early Detection (WRED) Pass Transmit Queue
Discard Test
Output Line
Fail
FIFO Scheduling
Discard Based On: • Buffer Queue Depth • QoS Assignment 502 0943_05F9_c2
Interface Buffers
49
© 1999, Cisco Systems, Inc.
Link Management: Increased Link Utilization Data from a Burst E1 (2.0 Mbps) Courtesy of Sean Doran RED Was Turned on Friday at 10:00 am; Link Utilization Goes Up to Near 100%
Thursday 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
Friday
50
25
Link Management (Cont.) Link Utilization Went from about 1700 kbps to 2.0 Mbps Average TCP Window Size Is Higher; Multiple Packets from Same Window Are Not Dropped
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
51
Other Benefits of RED
• Beyond the higher link utilization other benefits that are not graphically shown include: Small users are not negatively biased Fewer retransmissions because there are fewer restarts 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
52
26
Policing Action • Device (switch/router) has a contract with the Network, e.g., only allow 3M of HTTP traffic • Rest is discarded (e.g., Token Bucket) or marked down from the current value • A new label is assigned based on the configuration 502 0943_05F9_c2
53
© 1999, Cisco Systems, Inc.
Token Bucket for Policing • Start with a bucket full of tokens • Tokens can be removed at a bursty rate
Average Rate
Max Burst
• Tokens are replaced at a specified constant rate 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
Bursts
54
27
Re-Classification • CoS can also be assigned by the layer 2 forwarding engine The user can configure mac addresses to be associated with a CoS When packets arrive destined to such a mac address, their CoS is re-assigned to the value specified
• Assignment of CoS via dest-mac overrides trust and port-based CoS assignment 502 0943_05F9_c2
55
© 1999, Cisco Systems, Inc.
Output Scheduling • On output, there are multiple queues, some high priority and some low priority • Strict priority servicing of the queue would cause starvation of low priority traffic in the presence of high priority traffic • However, delay requirements of high priority traffic must also be respected 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
56
28
Output Scheduling—WRR Weighted Round Robin Used to Schedule Between Queues
Low
Medium
High
WRR Queue Scheduler
Output Port 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
57
Weighted Round Robin
• The Network Administrator specifies weights associated with each of the queues The weight ranges from 1–255
• Link is shared in the proportion specified by the weights 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
58
29
Example with Dual Queues Two Queues with Four Drop Thresholds
Drop Priority
Service
Delay Insensitive
Delay Sensitive
Mission Critical
7
6
Premium Best Effort
5
4
Regular Best Effort
3
2
Discard Eligible
1
0
Delay Priority 502 0943_05F9_c2
59
© 1999, Cisco Systems, Inc.
QoS-Enabled Switching Wiring Closet (Access Layer)
Backbone (Distribution Layer)
WAN Edge (Core Layer)
QoS Ingress
QoS Core
QoS WAN Edge
Admission Control Classification Policing
Congestion Avoidance Congestion Mgmt Scheduling
Congestion Avoidance Congestion Mgmt Policing, Shaping, Fragmentation, Header Compression
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
60
30
Agenda • Networking Trends • QoS Demystified • QoS in Campus: Justifications • QoS Model and Techniques • Implementing QoS • Summary • Q&A 502 0943_05F9_c2
61
© 1999, Cisco Systems, Inc.
Example 1: Prioritization of IP Telephony
Set Telephony = High TOS = 7 Set Game = Low TOS = 2
Si
For TOS = 7 Threshold = 4 High Priority Queue (70% Transmit Ratio, Low Delay)
For TOS = 2 Threshold = 2 Low Priority Queue (30% Transmit Ratio, High Delay) 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
Si
62
31
Example 2: ERP Application Untrusted Client
QOS Ingress
QOS Core
Reclassify Schedule TCP, L4-Port=1521 According Set TOS=7 to TOS =7 Drop Threshold=Low SQL Client
Access Switch
Database 10.1.2.1
Backbone Switch
Database 10.1.2.2
Server Farm Switch
SQL Listener Server Farm
502 0943_05F9_c2
Schedule According to TOS =7
Reclassify IF: L4-Port=1521 IP-SA/DA = 10.1.2.1 IP-SA/DA = 10.1.2.2 Set TOS=7
QOS Core
QOS Ingress
Client 63
© 1999, Cisco Systems, Inc.
Example 3: SNA Traffic L2 ISL or 802.1p Class of Service Reclassification
Reclassify IF: DA-MAC = MAC of FEP
Cluster Controller
Cat 5k
Port 2065 1984 1983 1981 1981
DLSW
L2 Connection
Optional DLSW cloud
Cat 6k FEP and Mainframe
Reclassify All Traffic on FEP Physical Port 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
64
32
Agenda • Networking Trends • QoS Demystified • QoS in Campus: Justifications • QoS Model and Techniques • Implementing QoS • Summary • Q&A 502 0943_05F9_c2
65
© 1999, Cisco Systems, Inc.
Conclusions • QoS in campus is: Meaningful Important Needed
• Campus (LAN) QoS requirements are emerging • Prerequisite to D/V/V deployment, is the need to prioritize mission-critical applications 502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
66
33
Agenda • Networking Trends • QoS Demystified • QoS in Campus: Justifications • QoS Model and Techniques • Implementing QoS • Summary • Q&A 502 0943_05F9_c2
67
© 1999, Cisco Systems, Inc.
Please Complete Your Evaluation Form Session 502
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
68
34
502 0943_05F9_c2
© 1999, Cisco Systems, Inc.
Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. Presentation_ID.scr
69
35
