Cisco - Establishing Best Practices for Network Management 804

804 1066_05F9_c2

1

© 1999, Cisco Systems, Inc.

Establishing Best Practices for Network Management Session 804

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

2

1

Agenda • • • • •

804 1066_05F9_c2

Introduction to Best Practices Preparing the Network for Management Managing Change Fault Management Summary

3

© 1999, Cisco Systems, Inc.

Introduction to Best Practices

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

4

2

Network Downtime is Costly • The Internet and e-commerce has significantly increased the availability stakes… 24-hour banking E-trade Global economy

8

Infonetics Cost of WAN Downtime ’98

7 6 5

Average Dollars 4 per Year 3 ($000,000)

2 1 0

$3.6M Productivity Productivity Loss Loss

$4.2M

$3.6M

Revenue Revenue Loss Loss

Costs

Enterprise Network Mgmt. Budget

*Due to hard downtime and service degradations 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

5

Best Practices Defined • Applying what works well for others to improve overall network availability Reduce the time required for planned outages (scheduled change) and includes changes with no associated outage Reduce network downtime during unplanned outages (unscheduled change)

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

6

3

Lots of Practices—Some Truths • Even the best NM products can be useless with “bad” practices

Do What Works for You!

• Tools help you to do your job, they are NOT the job • Communication and security are the “bread and butter” of best practices 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

7

Preparing the Network

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

8

4

Congratulations! You’ve just Been Promoted to Manage the Entire Network for the Western Region...

804 1066_05F9_c2

9

© 1999, Cisco Systems, Inc.

What They’re Really Thinking…

What am I getting into… how am I going to do this? Where do I begin?

804 1066_05F9_c2

I sure hope he lasts longer than the last guy..

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

What a loser! Does he have any idea what he’s in for? How come we don’t have legs?

10

5

Preparing the Network for Management Best Practices 1. 2. 3. 4. 5.

804 1066_05F9_c2

Selecting the “right” tools Preparing the devices Preparing the tools Building a baseline Maintaining “management”

11

© 1999, Cisco Systems, Inc.

Selecting the Right Tools

? • How do I select the “right” set of management applications? Understand the technologies and buzzwords Understand your network and end-user requirements Implement company standards Many choices evaluate and choose what’s right for your environment 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

12

6

Platforms and Vendor Specific Management • NMS SNMP-based, status map, and trap receiver HP Openview, Tivoli Netview, CA UniCenter, SNMPc, etc. MicroMuse, Seagate, Concord, Enterprise Pro, and MRTG

• Vendor Specific Geared towards managing a specific vendors devices only Optivity, Transcend, CiscoWorks2000

804 1066_05F9_c2

13

© 1999, Cisco Systems, Inc.

Integrating Enterprise Management Helpdesk, Trouble-ticket, Event MOM

Application Application

DBMS DBMS

Server Server

Service

804 1066_05F9_c2

Service

Service

Network Network

Device

Device

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

Device

Desktop Desktop

Device

Device

User User

Device

14

7

Understand Your Organization • Roles and responsibilities • Escalation policy • Help desk vs. operations • Planners vs. administrators 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

15

Preparing the Devices

• Security for Management • Notification • Baseline 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

16

8

Securing the Devices • Identify scope of control Who needs access to what?

• Secure and log access Physical access (badge readers) Telnet and console (AAA accounting, Syslog) SNMP communities (ACL, SNMP traps) 804 1066_05F9_c2

17

© 1999, Cisco Systems, Inc.

Sample Security Configuration

aaa new-model aaa authentication login test tacacs+ line aaa authentication enable default tacacs+ enable access-list 8 permit 161.44.34.157 logging 161.44.34.157 logging source-interface Loopback0 snmp-server community public RO snmp-server community bitbuck RW 8 snmp-server contact Paul L. Della Maggiora snmp-server chassis-id 071293 snmp-server system-shutdown snmp-server trap-source Loopback0 snmp-server trap-authentication snmp-server host 161.44.34.157 public frame-relay

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

Tacacs+ SNMP Community ACL Syslog SNMP gets and sets

SNMP traps

18

9

Security Access Changes

• Password change policy Quarterly Every time an employee leaves

• Solution Use radius or tacacs+ Script the change 804 1066_05F9_c2

19

© 1999, Cisco Systems, Inc.

Notification • SNMP Traps Critical for NMS notification

• Syslog Cisco-specific notification

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

20

10

Sample Notification Configuration

aaa new-model aaa authentication login test tacacs+ line aaa authentication enable default tacacs+ enable access-list 8 permit 161.44.34.157 logging 161.44.34.157 logging source-interface Loopback0 snmp-server community public RO snmp-server community bitbuck RW 8 snmp-server contact Paul L. Della Maggiora snmp-server chassis-id 071293 snmp-server system-shutdown snmp-server trap-source Loopback0 snmp-server trap-authentication snmp-server host 161.44.34.157 public frame-relay

804 1066_05F9_c2

Tacacs+ SNMP Community ACL Syslog SNMP gets and sets

SNMP traps

21

© 1999, Cisco Systems, Inc.

Building a Baseline

• Document the network Maps Spreadsheets/databases

• Track inventory Identify equipment and who owns it

• Backup configurations 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

22

11

Building a Baseline

• Collect performance data Snapshot of the network Provides historical data for comparison Useful for capacity planning and trending

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

23

Discovering the Network • Auto-discovery can make documentation easy… but the daemons must be tamed Filters Seedfiles Discovery intervals Exchange inventory among multiple autodiscovery tools 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

24

12

Layer 2 Autodiscovery 1. Query seed device via SNMP 2. Query CDP neighbor table (ciscoCdpMIBObjects) 3. Interrogate neighbors Caveat—CDP only sees Cisco devices c55k-26 (enable) sho cdp neigh Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater Port Device-ID Capability -------- ------------------------4/1 002261261 4/1 002274433 4/1 069004796 4/1 Router_81.130 4/1 WBU_GATEWAY

804 1066_05F9_c2

Port-ID

Platform

----------------- ------------------ ------4/1 4/1 4/1 Ethernet0 Ethernet0

WS-C5000 WS-C5000 WS-C5500 cisco 4500 cisco 4500

T B S T B S T B S R R

© 1999, Cisco Systems, Inc.

25

Layer 3 Autodiscovery 1. Start with default router 2. Query MIB II ifTable, ipAddrTable, ipRouteTable 3. Interrogate neighbors Special cases e.g. IP unnumbered, HSRP 4500-4>sho ip rout Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, * - candidate default U - per-user static route Gateway of last resort is not set

O C C 804 1066_05F9_c2

100.0.0.0/8 is subnetted, 1 subnets 100.100.100.0 [110/70] via 172.16.11.1, 13:35:34, Serial0 153.10.0.0/16 is subnetted, 1 subnets 153.10.1.0 is directly connected, Serial1 172.16.0.0/16 is subnetted, 1 subnets 172.16.11.0 is directly connected, Serial0

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

26

13

Inventory

• Typical NMS is not enough IP address, comm strings, and interfaces

• Third-party management suites and vendor specific provide richer content • MIBs are generally vendor specific, although entity MIB will change this 804 1066_05F9_c2

27

© 1999, Cisco Systems, Inc.

Inventory • Items of interest System information Chassis information Chassis cards Interfaces Storage and memory Serial numbers

• All information available via IETF and Cisco MIBs 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

28

14

Configurations • Collection repository Useful for staging new configs Version control helps with space and documentation

• How to automate Scheduled backup Watch Syslog 804 1066_05F9_c2

29

© 1999, Cisco Systems, Inc.

Maintaining Management

• Adding new devices • Keeping the management applications up-to-date • New management products and standards An Ongoing Process! 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

30

15

Change Management

804 1066_05F9_c2

31

© 1999, Cisco Systems, Inc.

Post Mortem Blues • Unplanned outages may be the result of many factors. How do you explain and account for what occurred?

I Didn’t Do It

Fact based vs. hearsay Who, what, and when was the change made? Your job may be at stake 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

32

16

Some Facts

• 80% of all outages are due to human error* When an airlines reservation system went down, thousands of travel agents had to book flights manually. Estimated loss of reservations amounted to $36,000 a minute

*Based on Carnegie-Mellon Usability Study 804 1066_05F9_c2

X

© 1999, Cisco Systems, Inc.

33

Common Causes of Change

• Business growth or downsizing • New applications or services • Implementing new technology • Deploying product fixes or upgrades

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

34

17

Change Management Defined

• Configuration, software and hardware changes • Change tasks include: Anticipating and planning for change, controlling the introduction of change, and installing and implementing changes to software and hardware 804 1066_05F9_c2

35

© 1999, Cisco Systems, Inc.

Best Practices for Change Best Practices 1. Implementing a change control process 2. Planning for change 3. Implementing change 4. Monitoring change

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

36

18

Change Control Process

Change request • End user request • New app, server • New network service

Change review board • Identify risk • Schedule change • Generate work order

Change or work order • Tracking # • Detailed change requests

Close Work Order or Resubmit If Problems

Validation • Change verification • Audit

Implementation • Net admin • Engineer/tech.

804 1066_05F9_c2

37

© 1999, Cisco Systems, Inc.

Examples

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

38

19

Planning • Hardware Pre-configure, test prior to upgrade

• Software Research release, defect support, new feature set, and device compatibility

• Configuration Test prior to deployment

• Have a back-out plan 804 1066_05F9_c2

39

© 1999, Cisco Systems, Inc.

Implementing

• Make different types of changes one at a time • Maker/checker model • Understand contingency plan in event of failure • Validate the change was successful 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

40

20

Monitoring

• Identifying change, who, what, when • Audit trail • Fault notification

804 1066_05F9_c2

41

© 1999, Cisco Systems, Inc.

Change Management Tools Planning SWIM—Defect, SWIM—Defect, image image analysis analysis CWSI—Layer CWSI—Layer 2/Layer 2/Layer 33 topo topo Netsys—Impact Netsys—Impact of of change change

Deployment

Monitor

SWIM—Download SWIM—Download software software images images CWConfig—Deploy CWConfig—Deploy config config changes changes CiscoView—Switch CiscoView—Switch config config changes changes

CAS—Change CAS—Change audit audit and and reporting reporting service, service, logs logs software, software, config config and and hardware hardware changes changes CWSI—Topo CWSI—Topo and and user user tracking tracking

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

42

21

Change Scenario 1. User telnets into device and makes a config change (shutdown int) 3. C/Agent identifies device change, notifies archive

Server

Change Agent

4. Archive gets config via transport validates change w/DIFF Archive Audit Log

5. IF VALID, Archive gets Config and logs details to ENCASE Syslog Poll

Transport Change

Network 804 1066_05F9_c2

2. Device updated Syslog generated

© 1999, Cisco Systems, Inc.

43

Fault Management

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

44

22

Scenario

• Virginia building-003 network goes down • Your boss has bad breath • Multiple people making changes • Resolution takes nine hours 804 1066_05F9_c2

45

© 1999, Cisco Systems, Inc.

Scenario

• Result: Network was down additional four hours due to conflicting changes No one seems to know how the problem occurred or how it was resolved

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

46

23

Best Practices for Fault Management Best Practices 1. 2. 3. 4. 4.

804 1066_05F9_c2

Preventive Measures Coordination Reacting to Faults Escalation Policy Become Proactive

© 1999, Cisco Systems, Inc.

47

Preventive Measures • Maintain accurate documentation Key to quick resolution Includes maps, closets, connections, wiring, and servers May require process/policy change. Only good if up to date, easy to maintain, and useful Dump it if you can’t maintain it! 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

48

24

Preventive Measures • Remove single points of failure Alternate paths for mission-critical applications Redundant equipment for critical junctures Ensure appropriate bandwidth to avoid contention and over utilization Permits network rerouting 804 1066_05F9_c2

49

© 1999, Cisco Systems, Inc.

Coordination • Communication is KEY... Understand roles and responsibilities Place phones in closets; use cell phones, pagers Publish policies and procedures 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

Say What You Do, Do What You Say 50

25

Coordination

• Establish base of operations All efforts must go through one person Prevents “who dropped the baby” and “slam management” Conduct practice “scramble”

• Train staff on devices and technology 804 1066_05F9_c2

51

© 1999, Cisco Systems, Inc.

Determination of Faults

• Notification via: NMS status change Trap and event logs Help desk Phone call from tech (“whoops...”) 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

ALARM

52

26

Determination of Faults

• Remove the “noise” factor 1. Filter 2. Prioritize 3. Appropriately notify 4. Correlate 804 1066_05F9_c2

53

© 1999, Cisco Systems, Inc.

Reacting to Faults • Determine fault domain Which equipment, services, and users are affected?

• Determine level of response What is the severity of the fault? Can we kill the backbone? Identify dispatch timeframe and number of people 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

54

27

Reacting to Faults (Severe) • Determine escalation timeline Criteria and time limits to escalate to next level Opening a case with the TAC Identifying the point of drastic action 804 1066_05F9_c2

Is It Time to Hit the Big Red Switch?

© 1999, Cisco Systems, Inc.

55

Reacting to Faults (severe) • Coordinate, communicate, and document • Debrief Determine source of fault Evaluate recovery efforts Document resolution for continuous improvement process In order to learn, avoid CYA environment 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

56

28

Moving from Reactive to Proactive

• Automate fault notification, escalation and resolution via “triggers” • React to data before it goes bad • Learn device and network behavior That doesn’t look right… 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

57

Active vs. Passive Polling

• Polling with thresholds vs. event-based polling RMON events and alarms

• Conservation of network traffic vs. device CPU and memory • Might be a combination of both 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

58

29

Fault Management Tools Planning CiscoView— CiscoView— Real-time Real-time time time monitoring monitoring RME—Availability, RME—Availability, Syslog Syslog and and CCO CCO tools tools CWSI—User CWSI—User tracking, tracking, traffic traffic director director and and topo topo

Deployment SWIM— SWIM— Defect Defect analysis analysis CCO/TAC— CCO/TAC— Case Case tracking tracking tools tools Stack Stack Decoder— Decoder— Crash Crash analysis analysis

804 1066_05F9_c2

Monitor Availability— Availability— Monitor Monitor key key resources resources Syslog—Reporting, Syslog—Reporting, automated automated recovery recovery 24-Hour 24-Hour Reports— Reports— Monitor Monitor reloads, reloads, Syslog, Syslog, and and change change Traffic Traffic Director—RMON Director—RMON config config and and report report

© 1999, Cisco Systems, Inc.

59

Best Practices Can Improve Network Availability • Prepare the network for management Security, notification and maintenance

• Implement a change control process Plan, deploy and monitor

• Reduce unplanned outage minutes through fault management Prepare, coordinate and be proactive 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

60

30

For More Information • General network management portal http://netman.cit.buffalo.edu/index.html

• Another good network management portal http://compnetworking.miningco.com/msubmanage.htm? terms=network+management&cob=home&TMog= 5006366091143m&Mint=56534342191358&FFV=1

• “The Simple Times” http://www.simple-times.org/pub/simple-times/issues/

• SNMP FAQ http://www.cis.ohio-state.edu/hypertext/faq/usenet/ snmp-faq/part1/faq.html 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

61

For More Information • Sample Cisco device security configs http://www.cisco.com/warp/public/700/tech_configs .html#SECURITY

• Cisco device SNMP configuration tips http://www.cisco.com/warp/public/490/index.shtml

• White paper on threshold management http://www.ccci.com/product/papers/pete/papers/thresh.htm

• Public domain performance monitoring tool (MRTG) http://ee-staff.ethz.ch/~oetiker/webtools/mrtg/mrtg.html 804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

62

31

Please Complete Your Evaluation Form Session 804

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

63

804 1066_05F9_c2

© 1999, Cisco Systems, Inc.

64

Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

32