Front cover
Tuning IBM System x Servers for Performance Identify and eliminate performance bottlenecks in key subsystems Expert knowledge from inside the IBM performance labs Covers Windows, Linux, and ESX Server
David Watts Erwan Auffret Phillip Dundas Mark Kapoor Daniel Koeck Charles Stephan Foreword by Gregg McKnight
ibm.com/redbooks
International Technical Support Organization Tuning IBM System x Servers for Performance February 2007
SG24-5287-04
Note: Before using this information and the product it supports, read the information in “Notices” on page xix.
Fifth Edition (February 2007) This edition applies to IBM System x servers running Windows Server 2003, Red Hat Enterprise Linux, SUSE Linux Enterprise Server, and VMware ESX Server.
© Copyright International Business Machines Corporation 1998, 2000, 2002, 2004, 2007. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
Contents Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii The team that wrote this edition of the IBM Redbook . . . . . . . . . . . . . . . . . . xxiv Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxix Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxix Part 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Chapter 1. Introduction to this IBM Redbook . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 Operating an efficient server—four phases. . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Performance tuning guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 The System x Performance Lab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.4 IBM Center for Microsoft Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.5 Linux Technology Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.6 Understanding the organization of this IBM Redbook . . . . . . . . . . . . . . . . . 8 Chapter 2. Understanding server types . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1 Server scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2 Authentication services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.2.1 Windows Server 2003 R2 Active Directory domain controllers . . . . . 14 2.3 File servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.4 Print servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.5 Database servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.6 E-mail servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.7 Web servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.8 Groupware servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.9 Multimedia server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.10 Communication server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.11 Terminal server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.12 Infrastructure servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.12.1 DNS server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.12.2 DHCP server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.12.3 WINS server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.13 Virtualization servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.14 High Performance Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Contents
iii
Part 2. Server subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Chapter 3. Introduction to hardware technology . . . . . . . . . . . . . . . . . . . . 31 3.1 Server subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Chapter 4. Processors and cache subsystem . . . . . . . . . . . . . . . . . . . . . . 35 4.1 Processor technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.1.1 Single core Intel Xeon processors . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.1.2 Dual core Intel Xeon processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.1.3 Quad core Intel Xeon processors . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.1.4 Intel Core microarchitecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.1.5 Opteron processors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.1.6 Itanium 2 processors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.2 64-bit computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.3 Processor performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.3.1 Comparing CPU architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.3.2 Cache associativity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.3.3 Cache size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4.3.4 CPU clock speed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.3.5 Scaling versus the number of processor cores . . . . . . . . . . . . . . . . . 72 4.3.6 Processor features in BIOS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.4 Rules of thumb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 Chapter 5. Virtualization hardware assists . . . . . . . . . . . . . . . . . . . . . . . . . 79 5.1 Introduction to virtualization technology . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.1.1 Privilege levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.1.2 Binary translation and paravirtualization . . . . . . . . . . . . . . . . . . . . . . 81 5.2 Virtualization hardware assists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.3 Support for virtualization hardware assists . . . . . . . . . . . . . . . . . . . . . . . . 84 Chapter 6. PCI bus subsystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 6.1 PCI and PCI-X. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.2 PCI-X. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.2.1 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 6.3 PCI Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 6.3.1 PCI Express performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.4 Bridges and buses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Chapter 7. Chipset architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 7.1 Overview of chipsets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 7.2 System architecture design and performance . . . . . . . . . . . . . . . . . . . . . 100 7.2.1 Hardware scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 7.2.2 SMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 7.2.3 NUMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
iv
Tuning IBM System x Servers for Performance
7.2.4 The MESI protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 7.2.5 Software scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 7.3 Memory controller-based chipset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 7.3.1 ServerWorks Grand Champion 4.0 HE and LE. . . . . . . . . . . . . . . . 111 7.3.2 Intel E7520 and E7525 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 7.3.3 Intel 5000 chipset family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 7.3.4 XA-64e third generation chipset . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 7.3.5 Intel E8500 Chipset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 7.4 PCI bridge-based chipsets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Chapter 8. Memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 8.1 Memory technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 8.1.1 DIMMs and DRAMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 8.1.2 Ranks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 8.1.3 SDRAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 8.1.4 Registered and unbuffered DIMMs . . . . . . . . . . . . . . . . . . . . . . . . . 134 8.1.5 Double Data Rate memory, DDR and DDR2 . . . . . . . . . . . . . . . . . 135 8.1.6 Fully-buffered DIMMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 8.1.7 DIMM nomenclature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 8.1.8 DIMMs layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 8.1.9 Memory interleaving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 8.2 Specifying memory performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 8.2.1 Bandwidth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 8.2.2 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 8.2.3 Loaded versus unloaded latency . . . . . . . . . . . . . . . . . . . . . . . . . . 146 8.2.4 STREAM benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 8.3 SMP and NUMA architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 8.3.1 SMP architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 8.3.2 NUMA architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 8.4 The 32-bit 4 GB memory limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 8.4.1 Physical Address Extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 8.5 64-bit memory addressing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 8.6 Advanced ECC memory (Chipkill) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 8.7 Memory mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 8.8 X3 architecture servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 8.9 IBM Xcelerated Memory Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 8.10 BIOS levels and DIMM placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 8.11 Memory rules of thumb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Chapter 9. Disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 9.2 Disk array controller operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 9.3 Direct Attached Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Contents
v
9.3.1 SAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 9.3.2 Serial ATA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 9.4 Remote storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 9.4.1 Differences between SAN and NAS . . . . . . . . . . . . . . . . . . . . . . . . 182 9.4.2 Fibre Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 9.4.3 iSCSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 9.5 RAID summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 9.5.1 RAID-0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 9.5.2 RAID-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 9.5.3 RAID-1E . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 9.5.4 RAID-4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 9.5.5 RAID-5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 9.5.6 RAID-5EE and RAID-5E . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 9.5.7 RAID-6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 9.5.8 Composite RAID levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 9.6 Factors that affect disk performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 9.6.1 RAID strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 9.6.2 Number of drives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 9.6.3 Active data set size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 9.6.4 Drive performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 9.6.5 Logical drive configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 9.6.6 Stripe size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 9.6.7 SCSI bus organization and speed. . . . . . . . . . . . . . . . . . . . . . . . . . 213 9.6.8 Disk cache write-back versus write-through . . . . . . . . . . . . . . . . . . 214 9.6.9 RAID adapter cache size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 9.6.10 Rebuild time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 9.6.11 Device drivers and firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 9.6.12 Fibre Channel performance considerations . . . . . . . . . . . . . . . . . 220 9.7 Disk subsystem rules of thumb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 9.8 Tuning with IBM DS4000 Storage Manager . . . . . . . . . . . . . . . . . . . . . . 229 Chapter 10. Network subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 10.1 LAN operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 10.1.1 LAN and TCP/IP performance . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 10.2 Factors affecting network controller performance . . . . . . . . . . . . . . . . . 242 10.2.1 Transfer size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 10.2.2 Number of Ethernet ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 10.2.3 CPU and front-side bus. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 10.2.4 Jumbo frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 10.2.5 10 Gigabit Ethernet adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 10.2.6 LAN subsystem performance summary . . . . . . . . . . . . . . . . . . . . 257 10.3 Advanced network features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 10.3.1 TCP offload engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
vi
Tuning IBM System x Servers for Performance
10.3.2 I/O Accelerator Technology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 10.3.3 Comparing TOE and I/OAT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 10.3.4 TCP Chimney Offload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 10.3.5 Receive-side scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 10.3.6 RDMA overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 10.3.7 Operating system considerations . . . . . . . . . . . . . . . . . . . . . . . . . 282 10.4 Internet SCSI (iSCSI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 10.4.1 iSCSI Initiators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 10.4.2 iSCSI network infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 10.5 Interconnects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 10.5.1 Myrinet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 10.5.2 InfiniBand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 Part 3. Operating systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 Chapter 11. Microsoft Windows Server. . . . . . . . . . . . . . . . . . . . . . . . . . . 295 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 11.2 Windows Server 2003, 64-bit (x64) Editions . . . . . . . . . . . . . . . . . . . . . 298 11.2.1 32-bit limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 11.2.2 64-bit Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 11.2.3 The transition to 64-bit computing . . . . . . . . . . . . . . . . . . . . . . . . . 301 11.2.4 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 11.3 Windows Server 2003, Release 2 (R2) . . . . . . . . . . . . . . . . . . . . . . . . . 302 11.4 Processor scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 11.5 Virtual memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 11.5.1 Configuring the pagefile for maximum performance gain . . . . . . . 307 11.5.2 Creating the pagefile to optimize performance . . . . . . . . . . . . . . . 308 11.5.3 Measuring pagefile usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 11.6 File system cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 11.6.1 Servers with large amounts of free physical memory . . . . . . . . . . 314 11.7 Disabling or removing unnecessary services . . . . . . . . . . . . . . . . . . . . 315 11.8 Removing unnecessary protocols and services . . . . . . . . . . . . . . . . . . 318 11.9 Optimizing the protocol binding and provider order. . . . . . . . . . . . . . . . 320 11.10 Optimizing network card settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322 11.11 Process scheduling, priority levels, and affinity. . . . . . . . . . . . . . . . . . 326 11.11.1 Process affinity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 11.12 Assigning interrupt affinity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 11.13 The /3GB BOOT.INI parameter (32-bit x86) . . . . . . . . . . . . . . . . . . . . 334 11.14 Using PAE and AWE to access memory above 4 GB (32-bit x86) . . . 335 11.14.1 Interaction of the /3GB and /PAE switches . . . . . . . . . . . . . . . . . 337 11.15 TCP/IP registry optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338 11.15.1 TCP window size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 11.15.2 Large TCP window scaling and RTT estimation (timestamps) . . 340
Contents
vii
11.15.3 TCP connection retransmissions . . . . . . . . . . . . . . . . . . . . . . . . 342 11.15.4 TCP data retransmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 11.15.5 TCP TIME-WAIT delay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 11.15.6 TCP Control Block (TCB) table . . . . . . . . . . . . . . . . . . . . . . . . . . 344 11.15.7 TCP acknowledgement frequency . . . . . . . . . . . . . . . . . . . . . . . 346 11.15.8 Maximum transmission unit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 11.15.9 Path Maximum Transmission Unit (PMTU) Discovery . . . . . . . . 349 11.16 Memory registry optimizations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 11.16.1 Disable kernel paging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 11.16.2 Optimizing the Paged Pool Size (32-bit x86) . . . . . . . . . . . . . . . 351 11.16.3 Increase memory available for I/O locking operations . . . . . . . . 353 11.16.4 Increasing available worker threads . . . . . . . . . . . . . . . . . . . . . . 354 11.16.5 Prevent the driver verifier from running randomly . . . . . . . . . . . . 356 11.17 File system optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 11.17.1 Increase work items and network control blocks. . . . . . . . . . . . . 356 11.17.2 Disable NTFS last access updates . . . . . . . . . . . . . . . . . . . . . . . 358 11.17.3 Disable short-file-name (8.3) generation . . . . . . . . . . . . . . . . . . . 359 11.17.4 Use NTFS on all volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 11.17.5 Do not use NTFS file compression . . . . . . . . . . . . . . . . . . . . . . . 360 11.17.6 Monitor drive space utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . 360 11.17.7 Use disk defragmentation tools regularly . . . . . . . . . . . . . . . . . . 361 11.17.8 Review disk controller stripe size and volume allocation units . . 361 11.17.9 Use auditing and encryption judiciously . . . . . . . . . . . . . . . . . . . 362 11.18 Other performance optimization techniques . . . . . . . . . . . . . . . . . . . . 363 11.18.1 Dedicate server roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 11.18.2 Run system intensive operations outside peak times . . . . . . . . . 363 11.18.3 Log off the server console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 11.18.4 Remove CPU-intensive screen savers . . . . . . . . . . . . . . . . . . . . 363 11.18.5 Use the latest drivers, firmware, and service packs . . . . . . . . . . 364 11.18.6 Avoid the use of NET SERVER CONFIG commands . . . . . . . . . 364 11.18.7 Monitor system performance appropriately . . . . . . . . . . . . . . . . . 367 11.19 The Future of Windows Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 11.19.1 Windows Server 2003, Service Pack 2 . . . . . . . . . . . . . . . . . . . . 368 11.19.2 Windows Server “Longhorn” . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368 11.19.3 The 64-logical thread limitation . . . . . . . . . . . . . . . . . . . . . . . . . . 370 Chapter 12. Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 12.1 Disabling daemons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 12.2 Shutting down the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 12.3 SELinux (RHEL 4 only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 12.4 Changing kernel parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 12.4.1 Parameter storage locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382 12.4.2 Using the sysctl commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
viii
Tuning IBM System x Servers for Performance
12.5 Kernel parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384 12.6 Tuning the processor subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388 12.6.1 Selecting the correct kernel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389 12.6.2 Interrupt handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 12.7 Tuning the memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 12.7.1 Configuring bdflush (kernel 2.4 only) . . . . . . . . . . . . . . . . . . . . . . 391 12.7.2 Configuring kswapd (kernel 2.4 only) . . . . . . . . . . . . . . . . . . . . . . 393 12.7.3 Setting kernel swap behavior (kernel 2.6 only) . . . . . . . . . . . . . . . 393 12.7.4 HugeTLBfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 12.8 Tuning the file system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 12.8.1 Hardware considerations before installing Linux. . . . . . . . . . . . . . 395 12.8.2 Ext3: the default Red Hat file system . . . . . . . . . . . . . . . . . . . . . . 397 12.8.3 ReiserFS: the default SUSE Linux file system . . . . . . . . . . . . . . . 398 12.8.4 File system tuning in the Linux kernel . . . . . . . . . . . . . . . . . . . . . . 398 12.8.5 The swap partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 12.9 Tuning the network subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407 12.9.1 Preventing a decrease in performance . . . . . . . . . . . . . . . . . . . . . 407 12.9.2 Tuning in TCP and UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408 12.10 SUSE Linux Enterprise Server 10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 12.10.1 Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 12.10.2 Administration and manageability . . . . . . . . . . . . . . . . . . . . . . . . 412 12.10.3 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 12.10.4 Scalability and performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414 12.10.5 Storage and high availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 12.10.6 Server services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 12.10.7 Application and developer services. . . . . . . . . . . . . . . . . . . . . . . 419 12.11 Xen virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420 12.11.1 What virtualization enables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420 12.11.2 Full virtualization versus paravirtualization . . . . . . . . . . . . . . . . . 421 12.11.3 CPU and memory virtualization. . . . . . . . . . . . . . . . . . . . . . . . . . 423 12.11.4 I/O virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424 Chapter 13. VMware ESX Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426 13.1.1 Understanding VMware performance concepts . . . . . . . . . . . . . . 426 13.2 General ESX Server tuning considerations. . . . . . . . . . . . . . . . . . . . . . 427 13.2.1 Hardware layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427 13.2.2 Manual NUMA tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430 13.2.3 VMware disk partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434 13.2.4 Tuning the VMware kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435 13.2.5 Tuning the virtual machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439 13.3 ESX Server 2.5 and later features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443 13.3.1 ESX Server 2.5 and later configuration best practices . . . . . . . . . 443
Contents
ix
13.3.2 The /proc file system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 13.3.3 Tuning the Console OS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448 13.3.4 ESX Server 2.5.x design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452 13.4 ESX Server 3.0 features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459 13.4.1 ESX 3.0 best practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 Part 4. Monitoring tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 Chapter 14. Windows tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471 14.1 Performance console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472 14.1.1 Comparing Performance console with Capacity Manager . . . . . . 472 14.1.2 Overview of the Performance console window . . . . . . . . . . . . . . . 474 14.1.3 Using System Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481 14.1.4 Using performance logs and alerts . . . . . . . . . . . . . . . . . . . . . . . . 486 14.2 Task Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505 14.2.1 Starting Task Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505 14.2.2 Processes tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506 14.2.3 Performance tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509 14.3 Network Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511 14.3.1 Installing Network Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512 14.3.2 Using Network Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513 14.4 Other Windows tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520 14.5 Windows Management Instrumentation . . . . . . . . . . . . . . . . . . . . . . . . 522 14.6 VTune . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528 Chapter 15. Linux tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537 15.1 The uptime command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539 15.2 The dmesg command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540 15.3 The top command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541 15.3.1 Process priority and nice levels. . . . . . . . . . . . . . . . . . . . . . . . . . . 542 15.3.2 Zombie processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543 15.4 The iostat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543 15.5 The vmstat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545 15.6 The sar command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546 15.7 KDE System Guard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547 15.7.1 The KSysguard work space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548 15.8 The free command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554 15.9 Traffic-vis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554 15.10 The pmap command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557 15.11 The strace command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558 15.12 The ulimit command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559 15.13 The mpstat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 560 15.14 System x Performance Logger for Linux . . . . . . . . . . . . . . . . . . . . . . . 561 15.14.1 Counters descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 562
x
Tuning IBM System x Servers for Performance
15.14.2 Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566 15.14.3 Parameter file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567 15.15 The nmon tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 15.15.1 Using nmon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572 15.15.2 The nmon Analyser Excel macro . . . . . . . . . . . . . . . . . . . . . . . . 576 Chapter 16. ESX Server tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 579 16.1 The esxtop utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580 16.1.1 Starting esxtop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581 16.1.2 Using esxtop with ESX Server 3.0 . . . . . . . . . . . . . . . . . . . . . . . . 585 16.1.3 Exiting esxtop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587 16.2 The vmkusage utility for performance monitoring . . . . . . . . . . . . . . . . . 588 Chapter 17. Capacity Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591 17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592 17.2 Capacity Manager data files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593 17.3 Installing Capacity Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594 17.4 Monitor Activator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596 17.5 Report Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597 17.5.1 Creating a new report definition . . . . . . . . . . . . . . . . . . . . . . . . . . 598 17.5.2 Working with predefined reports . . . . . . . . . . . . . . . . . . . . . . . . . . 604 17.5.3 Generating a report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605 17.6 Report Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607 17.6.1 Setting thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609 17.6.2 The System pane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611 17.6.3 The Monitor pane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613 17.6.4 The Graph pane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613 17.7 Performance analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617 17.7.1 Reports produced . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617 17.7.2 Types of bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 620 17.7.3 Setting critical and warning threshold values . . . . . . . . . . . . . . . . 620 17.7.4 Forecast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622 Part 5. Working with bottlenecks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625 Chapter 18. Spotting a bottleneck . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627 18.1 Achieving successful performance tuning . . . . . . . . . . . . . . . . . . . . . . . 629 18.2 Step 1: Gathering information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631 18.3 Step 2: Monitoring the server’s performance . . . . . . . . . . . . . . . . . . . . 633 18.3.1 Where to start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637 18.3.2 Disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 639 18.3.3 Memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645 18.3.4 Processor subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647 18.3.5 Network subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
Contents
xi
18.4 Step 3: Fixing the bottleneck . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651 18.5 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654 Chapter 19. Analyzing bottlenecks for servers running Windows . . . . . 655 19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656 19.2 CPU bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656 19.2.1 Finding CPU bottlenecks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657 19.2.2 Processor subsystem performance tuning options . . . . . . . . . . . . 660 19.3 Analyzing memory bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 661 19.3.1 Paged and non-paged RAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 662 19.3.2 Virtual memory system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 663 19.3.3 Performance tuning options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664 19.4 Disk bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667 19.4.1 Analyzing disk bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669 19.4.2 Performance tuning options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 670 19.5 Network bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 672 19.5.1 Finding network bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673 19.5.2 Analyzing network counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674 19.5.3 Solving network bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 678 19.5.4 Monitoring network protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684 Chapter 20. Analyzing bottlenecks for servers that are running Linux . 687 20.1 Identifying bottlenecks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 688 20.1.1 Gathering information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 688 20.1.2 Analyzing the server’s performance . . . . . . . . . . . . . . . . . . . . . . . 690 20.2 CPU bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 692 20.2.1 Finding bottlenecks with the CPU . . . . . . . . . . . . . . . . . . . . . . . . . 692 20.2.2 Multi-processing machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 693 20.2.3 Performance tuning options for the CPU . . . . . . . . . . . . . . . . . . . 693 20.3 Memory subsystem bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694 20.3.1 Finding bottlenecks in the memory subsystem . . . . . . . . . . . . . . . 694 20.3.2 Performance tuning options for the memory subsystem . . . . . . . . 697 20.4 Disk bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 698 20.4.1 Finding bottlenecks in the disk subsystem . . . . . . . . . . . . . . . . . . 698 20.4.2 Performance tuning options for the disk subsystem . . . . . . . . . . . 702 20.5 Network bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703 20.5.1 Finding network bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703 20.5.2 Performance tuning options for the network subsystem . . . . . . . . 704 Chapter 21. Case studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707 21.1 Analyzing systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 708 21.2 SQL Server database server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 709 21.2.1 Memory analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 709 21.2.2 Processor analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 711
xii
Tuning IBM System x Servers for Performance
21.2.3 Network analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 713 21.2.4 Disk analysis on the C: drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715 21.2.5 Disk analysis on the D: drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 717 21.2.6 Disk analysis of the V: drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 718 21.2.7 SQL Server analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 720 21.2.8 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 721 21.3 File servers hang for several seconds. . . . . . . . . . . . . . . . . . . . . . . . . . 722 21.3.1 Memory analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723 21.3.2 Processor analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724 21.3.3 Network analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725 21.3.4 Disks analysis of the V: drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . 726 21.3.5 System-level analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 728 21.4 Database server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 729 21.4.1 CPU subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 730 21.4.2 Memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 731 21.4.3 Disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 733 21.5 ERP application server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737 21.5.1 CPU subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737 21.5.2 Disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738 21.5.3 Memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 741 21.5.4 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 742 Part 6. Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 743 Chapter 22. File and print servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 745 22.1 File servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746 22.1.1 The effect of server hardware on performance . . . . . . . . . . . . . . . 747 22.1.2 Network subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 749 22.1.3 Disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 750 22.1.4 Tuning Windows Server 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 750 22.1.5 Tuning Linux: Samba . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 759 22.2 Print servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 761 Chapter 23. Lotus Domino . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 763 23.1 Performance tuning outside of Domino . . . . . . . . . . . . . . . . . . . . . . . . . 764 23.1.1 Network performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 764 23.1.2 Platform architecture and system performance . . . . . . . . . . . . . . 764 23.1.3 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 765 23.1.4 Disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 765 23.1.5 Network subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 768 23.2 Optimizing operating system performance . . . . . . . . . . . . . . . . . . . . . . 768 23.2.1 Operating system memory allocation . . . . . . . . . . . . . . . . . . . . . . 768 23.2.2 System cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 770 23.2.3 Application responsiveness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 772
Contents
xiii
23.2.4 Domino on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 773 23.3 Optimizing Lotus Domino performance . . . . . . . . . . . . . . . . . . . . . . . . . 774 23.3.1 Changing statements in the notes.ini file . . . . . . . . . . . . . . . . . . . 774 23.3.2 Configuring server tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775 23.3.3 Optimizing database performance . . . . . . . . . . . . . . . . . . . . . . . . 777 23.3.4 Defining the number of databases cached simultaneously . . . . . . 777 23.3.5 Scheduling utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 778 23.3.6 Optimizing database index update . . . . . . . . . . . . . . . . . . . . . . . . 779 23.3.7 Domino memory manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 787 23.3.8 Displaying images after documents . . . . . . . . . . . . . . . . . . . . . . . 789 23.3.9 Disabling the unread marks maintenance . . . . . . . . . . . . . . . . . . . 790 23.3.10 Optimizing of document table bitmap . . . . . . . . . . . . . . . . . . . . . 791 23.3.11 Do not maintain Accessed document property . . . . . . . . . . . . . . 792 23.3.12 Disabling specialized response hierarchy information . . . . . . . . 792 23.3.13 Preventing headline monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . 793 23.3.14 Limiting the number of entries in the $UpdatedBy field. . . . . . . . 793 23.3.15 Limiting the number of entries in the $Revisions field . . . . . . . . . 794 23.4 Improving mail performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 795 23.4.1 Setting maximum mail transfer threads. . . . . . . . . . . . . . . . . . . . . 795 23.4.2 Calendaring and scheduling resource use . . . . . . . . . . . . . . . . . . 801 23.4.3 Minimizing logging activity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 803 23.4.4 Improving Agent Manager performance . . . . . . . . . . . . . . . . . . . . 804 23.4.5 Managing server sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807 23.4.6 Controlling user access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 808 23.4.7 Improving replication performance . . . . . . . . . . . . . . . . . . . . . . . . 809 23.4.8 Enabling transaction logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 811 23.4.9 Improving Web server performance . . . . . . . . . . . . . . . . . . . . . . . 813 23.4.10 Network performance (compression) . . . . . . . . . . . . . . . . . . . . . 822 23.4.11 Using port encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825 23.4.12 Lotus Domino partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825 23.4.13 Balancing workload in a Lotus Domino cluster . . . . . . . . . . . . . . 826 23.5 Maintaining Lotus Domino servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 827 23.6 Planning for future growth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 828 23.7 Top 10 ways to improve Domino performance . . . . . . . . . . . . . . . . . . . 828 Chapter 24. Microsoft Exchange Server . . . . . . . . . . . . . . . . . . . . . . . . . . 831 24.1 Planning guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 832 24.2 Tuning guidelines for subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834 24.2.1 Network subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834 24.2.2 Memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 835 24.2.3 CPU. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 840 24.2.4 Disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 841 24.3 Exchange Server 2003 operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 843
xiv
Tuning IBM System x Servers for Performance
24.4 Exchange Server 2003 downloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . 844 24.5 Exchange 2007 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 845 Chapter 25. IBM DB2 Universal Database . . . . . . . . . . . . . . . . . . . . . . . . . 847 25.1 Optimizing the operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 848 25.2 CPU subsystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 849 25.2.1 Logical nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 849 25.2.2 Hyper-Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 850 25.2.3 Processor affinity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 850 25.2.4 Metrics to watch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 851 25.3 Memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 851 25.3.1 Metrics to watch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 855 25.4 Disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 855 25.4.1 Table spaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 856 25.4.2 Page size, extent size, and prefetch size . . . . . . . . . . . . . . . . . . . 857 25.4.3 Metrics to watch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 858 25.5 Networking and miscellaneous topics . . . . . . . . . . . . . . . . . . . . . . . . . . 858 Chapter 26. Microsoft SQL Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 861 26.1 Features of Microsoft SQL Server 2000 . . . . . . . . . . . . . . . . . . . . . . . . 862 26.1.1 SQL Server 2000 editions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 863 26.2 Features of SQL Server 2005 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 864 26.2.1 SQL Server 2005 editions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 865 26.2.2 New and enhanced features of SQL Server 2005 . . . . . . . . . . . . 865 26.2.3 Analysis Services enhancements . . . . . . . . . . . . . . . . . . . . . . . . . 867 26.2.4 Additional enhancements and features . . . . . . . . . . . . . . . . . . . . . 868 26.2.5 SQL Server Operating System . . . . . . . . . . . . . . . . . . . . . . . . . . . 869 26.3 Choosing which version of Windows to use . . . . . . . . . . . . . . . . . . . . . 870 26.3.1 Windows and SQL Server, both 32-bit . . . . . . . . . . . . . . . . . . . . . 871 26.3.2 Windows 64-bit and SQL Server 32-bit . . . . . . . . . . . . . . . . . . . . . 873 26.3.3 Windows and SQL Server 2005, both 64-bit . . . . . . . . . . . . . . . . . 873 26.4 The database environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874 26.5 SQL Server performance tuning basics . . . . . . . . . . . . . . . . . . . . . . . . 875 26.6 Server subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876 26.6.1 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876 26.6.2 Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 881 26.6.3 Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 884 26.6.4 Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 891 26.6.5 Hardware tuning versus application and database design . . . . . . 895 26.7 Scaling SQL Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 895 26.8 Optimizing Windows for SQL Server . . . . . . . . . . . . . . . . . . . . . . . . . . . 896 26.8.1 Processor scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 896 26.8.2 System cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 897
Contents
xv
26.8.3 Virtual memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 898 26.9 Further SQL Server optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 899 26.9.1 The max async I/O option (SQL Server 7.0 only) . . . . . . . . . . . . . 900 26.9.2 LazyWriter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 901 26.9.3 Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 902 26.9.4 Log manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 904 26.9.5 Read-ahead manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 905 26.9.6 Address Windowing Extensions support . . . . . . . . . . . . . . . . . . . . 906 26.10 SQL Server indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 908 26.10.1 Non-clustered indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 908 26.10.2 Clustered indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 909 26.10.3 Covering indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 910 26.10.4 Automatic covering indexes or covered queries . . . . . . . . . . . . . 910 26.10.5 Index selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 911 26.10.6 Clustered index selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 911 26.10.7 Importance of FILLFACTOR and PAD_INDEX . . . . . . . . . . . . . . 914 26.11 SQL Server performance objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 916 26.11.1 Other diagnostic and performance tools . . . . . . . . . . . . . . . . . . . 919 Chapter 27. Oracle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 923 27.1 Oracle architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 924 27.1.1 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 925 27.1.2 Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 927 27.1.3 DBMS files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 928 27.2 OLTP versus OLAP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 929 27.2.1 Online transaction processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 929 27.2.2 Online analytical processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 930 27.3 Important subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 931 27.4 Operating system optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 931 27.5 Oracle memory optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 932 27.5.1 Shared pool. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 933 27.5.2 Database buffer cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 933 27.5.3 Redo log buffer cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 937 27.6 Oracle disk I/O optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 937 27.6.1 RAID controllers cache size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 939 27.6.2 The optimal RAID level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 940 27.6.3 The optimal stripe unit size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 942 27.6.4 Oracle database block size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 943 27.6.5 Tablespace distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 943 27.6.6 Example configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945 27.7 Monitoring DBMS performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 946 27.8 Oracle Performance Monitor for Windows 2000 . . . . . . . . . . . . . . . . . . 947 27.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 949
xvi
Tuning IBM System x Servers for Performance
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 951 28.1 Thin-client or server-based computing . . . . . . . . . . . . . . . . . . . . . . . . . 952 28.2 Server sizing and scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 954 28.2.1 Scale up or scale out. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 954 28.2.2 Sizing considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 956 28.3 Server architecture and placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 958 28.3.1 User profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 960 28.4 Microsoft Windows Terminal Services . . . . . . . . . . . . . . . . . . . . . . . . . 961 28.4.1 Remote Administration versus Application Mode . . . . . . . . . . . . . 962 28.4.2 Remote Desktop Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 962 28.4.3 Tuning Windows Terminal Server . . . . . . . . . . . . . . . . . . . . . . . . . 964 28.5 Citrix Presentation Server 4.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 970 28.5.1 Presentation Server and Windows Terminal Services . . . . . . . . . 970 28.5.2 What’s new in Version 4.0. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 971 28.5.3 Performance features of Presentation Server 4.0 . . . . . . . . . . . . . 972 28.5.4 Tuning Citrix Presentation Server . . . . . . . . . . . . . . . . . . . . . . . . . 973 28.6 Common tuning options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 974 28.6.1 Remove Windows visual effects . . . . . . . . . . . . . . . . . . . . . . . . . . 975 28.6.2 TCP/IP keep alives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 976 28.6.3 Disable the System Tray network status icons . . . . . . . . . . . . . . . 977 28.7 Load balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 977 28.7.1 Network Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 977 28.7.2 Citrix Presentation Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 978 28.7.3 Load-balancing versus high-availability . . . . . . . . . . . . . . . . . . . . 979 28.8 Monitoring Windows Terminal Services . . . . . . . . . . . . . . . . . . . . . . . . 979 Chapter 29. Microsoft Internet Information Services . . . . . . . . . . . . . . . . 981 29.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 982 29.2 Tuning IIS 6.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 982 29.2.1 Worker process isolation mode . . . . . . . . . . . . . . . . . . . . . . . . . . . 983 29.2.2 Kernel mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 984 29.2.3 User mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 986 29.3 Hardware settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1002 29.3.1 Network subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1002 29.3.2 CPU subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1004 29.3.3 Other performance tuning factors . . . . . . . . . . . . . . . . . . . . . . . . 1008 29.4 Application tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1009 29.5 Monitoring performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1013 29.6 Network load balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1016 29.6.1 NLB with Layer 7 switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1020 Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1021
Contents
xvii
IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1021 Referenced Web sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1022 How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1030 IBM Redbooks collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1030 Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1031 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1039
xviii
Tuning IBM System x Servers for Performance
Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
xix
Trademarks The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: Redbooks (logo) ™ eServer™ ibm.com® iNotes™ xSeries® Active Memory™ AIX® BladeCenter® Chipkill™ Domino Designer® Domino® DB2 Universal Database™ DB2®
DFS™ DS4000™ ESCON® IBM® IMS™ Lotus Notes® Lotus® Netfinity Manager™ Netfinity® Notes® OS/2® OS/390® PowerPC®
POWER™ Redbooks™ RETAIN® ServeRAID™ System x™ System z™ System Storage™ Tivoli® TotalStorage® X-Architecture® Xcelerated Memory Technology™
The following terms are trademarks of other companies: Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of Oracle Corporation and/or its affiliates. ABAP, SAP R/3, SAP, and SAP logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries. Snapshot, and the Network Appliance logo are trademarks or registered trademarks of Network Appliance, Inc. in the U.S. and other countries. IPX, Java, JavaScript, JDK, JVM, J2EE, Solaris, Ultra, and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Active Desktop, Active Directory, BackOffice, Excel, Internet Explorer, Microsoft, MSDN, Outlook, PowerPoint, Visual Basic, Windows NT, Windows Server, Windows Vista, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Microsoft product screen shots reprinted with permission from Microsoft Corporation. i386, Intel SpeedStep, Intel, Itanium, MMX, Pentium, VTune, Xeon, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.
xx
Tuning IBM System x Servers for Performance
Foreword The genesis for this book began in 1997 when, in response to increasing customer demand for performance information, I decided to write a white paper addressing real-world performance issues. The title of that document was Fundamentals of Server Performance. This document was so well received by customers, business partners and IBM® support personnel that IBM decided to use it as the basis for a new Redbook addressing a multitude of real-world server performance issues. And in 1998 the Redbook Netfinity Performance Tuning with Windows NT 4.0 was published. Now in its fifth edition, Tuning IBM Systems x Servers for Performance is by far the most comprehensive and easy to understand performance guide specifically developed for Industry Standard servers. Yes Industry Standard servers, so if you deploy non-IBM servers you can also benefit greatly from this book. The explanations, tips and techniques can show you the way to better understanding server operation and solving even the most complex performance problems for any Windows or Linux®, Intel® or Opteron based server. In addition, this book will enlighten you on some of the special and unique performance optimizations IBM Engineers have introduced into IBM System x™ servers products. Finally, I would like to sincerely thank the team that wrote this latest version. Thank you for keeping this vital work current, informative and enjoyable to read. I’m certain the universe of server administrators and IT workers who benefit from the vast knowledge included in this volume also share my gratitude. Respectfully, Gregg McKnight Chief Technology Officer, Modular Systems Development IBM Distinguished Engineer IBM Corporation Research Triangle Park, North Carolina
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
xxi
xxii
Tuning IBM System x Servers for Performance
Preface This IBM Redbook describes what you can do to improve and maximize the performance of your business server applications running on IBM System x hardware and either Windows®, Linux or ESX Server operating systems. It describes how to improve the performance of the System x hardware, the operating system, and specific server applications such as Lotus® Domino®, Microsoft® SQL Server, and IBM DB2®. The keys to improving performance are to understand what configuration options are available to you and the monitoring tools that you can use and to analyze the results the tools provide so that you can implement suitable changes that positively affect the server. The book is divided into five parts. Part 1 explains the technology implemented in the major subsystems in System x servers and shows what settings can be selected or adjusted to obtain the best performance. Rules of thumb are supplied to give you advice about what to expect from any changes you consider. Each of the major subsystems covered in Part 1 are closely examined so that you can find specific bottlenecks and options are presented as to what can be done to resolve them. A discussion is provided to enable you to anticipate future bottlenecks as well. Part 2 describes the performance aspects of the operating systems Microsoft Windows Server® 2003, Red Hat Enterprise Linux, SUSE Linux Enterprise Server, and VMware ESX Server. Part 3 introduces the performance monitoring tools that are available to users of System x servers. We describe the tools specific to Windows, Linux and NetWare, as well as Capacity Manager, a component of IBM Director. Detailed instructions are provided showing you how to use these tools. Part 4 shows you how to analyze your system to find performance bottlenecks and what to do to eliminate them. We describe an approach you can take to solve a performance bottleneck. We also provide details about what to look for and how to resolve problems. Part 4 also includes a sample analysis of real-life servers, showing how tools can be used to detect bottlenecks and what the recommendations are for particular systems. Part 5 examines specific performance characteristics of specific server applications, including Lotus Notes®, SAP® R/3, DB2 UDB, SQL Server, Oracle® Database, Windows Terminal Services, IIS, and a typical file server. For
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
xxiii
each of these applications, the redbook explains what can be done to maximize the server’s performance. This book is targeted at people who configure Intel and AMD processor-based servers running Windows, Linux or ESX Server and seek to maximize performance. Some knowledge of servers is required. Skills in performance tuning are not assumed.
The team that wrote this edition of the IBM Redbook This edition of the redbook was produced by a team of specialists from around the world working at the International Technical Support Organization (ITSO), Raleigh Center. David Watts is a Consulting IT Specialist at the IBM ITSO Center in Raleigh. He manages residencies and produces Redbooks™ on hardware and software topics related to System x servers and associated client platforms. He has authored over 50 Redbooks and Redpapers. He has a Bachelor of Engineering degree from the University of Queensland (Australia) and has worked for IBM for over 15 years. He is an IBM Certified IT Specialist. Erwan Auffret is a System x IT specialist in Montpellier, France. His areas of expertise include IBM BladeCenter® and System x servers, IBM TotalStorage® subsystems, and Linux operating systems. He holds a master’s degree in computer sciences from the EFREI computers and electronics engineering school in Paris and has been working in the IT industry for four years. He works on customer performance benchmarks in the Products and Solutions Support Center of Montpellier and is part of the EMEA System x benchmarks center and the Network Transformation Center teams.
xxiv
Tuning IBM System x Servers for Performance
Phillip Dundas is the Technical Team Leader for the Windows Server Group at Macquarie Bank, based in Sydney, Australia. He has over 12 years of experience in network operating systems and server architecture, implementation, and support. His areas of technical expertise include Windows, VMWare, Citrix, Novell, server virtualization, server performance tuning, directory services, and establishing server standards. He also has extensive scripting experience and has developed several applications that are used by systems administrators to manage enterprise networks. Phillip holds a Bachelors degree in Applied Science (Computing) and a Masters degree in Commerce (Information Systems Management). Phillip holds many industry certifications, including Microsoft Certified Systems Engineer, VMWare Certified Professional, Certified Citrix Administrator, Master Certified Novell Engineer, and Certified Cisco Networking Associate. Mark Kapoor is an electrical engineer with IBM System x Modular Division in Research Triangle Park, North Carolina. He is currently team lead for technology validation in the high performance server group, concentrating on Enterprise X-Architecture® based systems. His academic background includes a Bachelor of Electrical Engineering degree from Purdue University and a Masters of Computer Engineering Degree from North Carolina State University. Prior to joining the high-end server development team, Mark spent five years on developing, optimizing, analyzing, modeling, and benchmarking high-end NUMA-based System x Servers for the TPC-C benchmark in the IBM System x Performance lab. Daniel Koeck is a System x Support Specialist in IBM Global Services in Austria. He has a graduate degree in applied computer science and has many industry certifications, including the Microsoft Certified Systems Engineer 2003. He trains other IBM professionals and provides technical support to them, as well as IBM Business Partners and customers. He has worked at IBM for four years, and his areas of expertise include Windows, networking, high-availability solutions, and software engineering.
Preface
xxv
Charles Stephan is a performance engineer on the System x Performance Team in Research Triangle Park, North Carolina. He has a Master of Science in Computer Information Systems from the Florida Institute of Technology. As a member of the System x Performance Team for the last nine years, he has worked on analyzing and optimizing the performance of ServeRAID™ SCSI products and DS4000™ Fibre Channel products. In addition, he is now analyzing and optimizing the performance of high-volume Intel and AMD servers, and high-performance Intel and AMD servers. Thanks to the authors of the previous editions of this IBM Redbook. Authors of the fourth edition, Tuning IBM eServer xSeries Servers for Performance, published in December 2004, were: David Watts Gregg McKnight Marcelo Baptista Martha Centeno Eduardo Ciliendo Jean-Jacques Clar Phillip Dundas Brian Jeffery Frank Pahor Raymond Phillips Luciano Tomé Authors of the third edition, Tuning IBM eServer xSeries Servers for Performance, published in July 2002, were: David Watts Gregg McKnight Jean-Jacques Clar Mauro Gatti Nils Heuer Karl Hohenauer Monty Wright Authors of the second edition, Tuning Netfinity Servers for Performance—Getting the Most Out of Windows 2000 and Windows NT, published by Prentice Hall in May 2000, were: David Watts Gregg McKnight Peter Mitura Chris Neophytou Murat Gülver
xxvi
Tuning IBM System x Servers for Performance
Authors of the first edition, Netfinity Performance Tuning with Windows NT 4.0, published in October 1998, were: David Watts Gregg McKnight M.S. Krishna Leonardo Tupaz Thanks also to the following people for their contributions in creating this IBM Redbook: IBM System x Performance Lab in Raleigh, NC: Payam Abrishami Ganesh Balakrishnan Martha Centeno Dustin Fredrickson Darryl Gardner Phil Horwitz Joe Jakubowski Marcus Kornegay Gregg McKnight Lily Shi Param Singh Charles Stephan Tricia Thomas Douglas Pase IBM System x Server Development team: Ralph Begun, Raleigh Maurice Bland, Raleigh Jim Hanna, Austin Nam Pham, Austin IBM technical and marketing specialists from around the world: Paul Branch, IBM Technical Alliance Manager for Microsoft, USA Jay Bretzmann, Worldwide System x Product Marketing Manager, USA Andreas Groth, Lead Engineer, ATS, Scotland Darryl Miles, IT Architect, Australia Kiron Rakkar, Worldwide AIM Lab Advocacy Program Manager, USA Massimo Re Ferre’, System x IT Specialist, Italy Leon Sienkiewicz, Senior IT Specialist, IBM Global Services, UK Bob Zuber, Worldwide Product Manager for System x, USA
Preface
xxvii
IBM ITSO personnel: Tamikia Barrow Byron Braswell Jere Cline Rufus Credle Cheryl Pecchia Linda Robinson Margaret Ticknor Jeanne Tucker Erica Wazewski Debbie Willmschen Lenovo: Roger Dodson Microsoft Corporation: Jim Katsandres Eric Keyser VMware: Jennifer Anderson Edouard Bugnion Anne Catambay John Hawkins Michael Mullany Peter Sonsini Red Hat: Nick Carr Pete Hnath Sandra Moore Arjan van de Ven SUSE Linux: Dennis Conrad Michael Hager Susanne Oberhauser Novell: Sue Goodwill Other contributors: Eran Yona
xxviii
Tuning IBM System x Servers for Performance
Become a published author Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners and customers. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html
Comments welcome Your comments are important to us! We want our Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at: ibm.com/redbooks Send your comments in an e-mail to:
[email protected] Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HYTD Mail Station P099 2455 South Road Poughkeepsie, NY 12601-5400
Preface
xxix
xxx
Tuning IBM System x Servers for Performance
Part 1
Part
1
Introduction
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
1
2
Tuning IBM System x Servers for Performance
1
Chapter 1.
Introduction to this IBM Redbook The server is the heart of the entire network operation. The performance of the server is a critical factor in the efficiency of the overall network and affects all users. While simply replacing the entire server with a newer and faster one might be an alternative, it is often more appropriate to replace or to add only to those components that need it and to leave the other components alone. Often, poor performance is due to bottlenecks in individual hardware subsystems, an incorrectly configured operating system, or a poorly tuned application. The proper tools can help you to diagnose these bottlenecks and removing them can help improve performance. For example, adding more memory or using the correct device driver can improve performance significantly. Sometimes, however, the hardware or software might not be the direct cause of the poor performance. Instead, the cause might be the way in which the server is configured. In this case, reconfiguring the server to suit the current needs might also lead to a considerable performance improvement.
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
3
This chapter provides an overall introduction to this IBM Redbook and discusses the following topics:
1.1, “Operating an efficient server—four phases” 1.2, “Performance tuning guidelines” on page 5 1.3, “The System x Performance Lab” on page 5 1.4, “IBM Center for Microsoft Technologies” on page 7 1.5, “Linux Technology Center” on page 7 1.6, “Understanding the organization of this IBM Redbook” on page 8
1.1 Operating an efficient server—four phases To operate an efficient server, you need to follow these four phases: 1. Have an overall understanding of the environment. There are many components within the network environment that can impact server performance and that can present themselves as potential bottlenecks. It is important to understand the role that the server has to play in this environment and to understand where it is located in the network and in relation to other servers in the environment. 2. Pick the correct server for the job. After you have established a need for a new server, it is important to have components that allow sufficient bandwidth through those critical subsystems. For example, a file server needs a disk subsystem and a network subsystem that provide sufficient bandwidth for client needs. 3. Configure the hardware appropriately and eliminate initial bottlenecks. After you have selected the server hardware (and application software), you need to configure the subsystems (for example, stripe size on the RAID array and RAID levels) to maximize performance. To ensure that you are actually improving performance, you need to take initial performance readings (called baseline readings) and then compare those with readings taken after you have implemented your changes. 4. Capture and analyze on-going performance data to ensure that bottlenecks do not occur. When the server is in production, you need to continue to gather and process performance figures to ensure that your server is still at a near-optimal configuration. You might need to add specific hardware upgrades, such as memory, to achieve this optimal configuration. As well as looking at the current situation, it is also appropriate that you perform trend analysis so that you can recognize future bottlenecks before
4
Tuning IBM System x Servers for Performance
they occur. Trend analysis allows you to plan for hardware upgrades before they are actually needed. Performance monitoring and tuning is an on-going task. It is not reasonable to simply tune a server once and then assume that it will remain tuned forever. Because the server workload mix changes, so do the location and appearance (and disappearance) of bottlenecks.
1.2 Performance tuning guidelines Table 1-1 lists guidelines to assist you with server management and performance tuning. Although not directly applicable to tuning, following these guidelines should assist you in preventing bottlenecks and identifying bottlenecks. Table 1-1 Performance tuning guidelines Guideline
Reasoning
Centralize servers where possible
Assists with management and can isolate components such as WAN.
Minimize the number of server types
Enables you to focus on specific subsystems within specific server types.
Standardize configurations
Enable you to focus on specific subsystems within specific server types.
Use industry accepted protocols and standards
Prevents attempts to identify bottlenecks with obscure third-party products and tools.
Use appropriate tools
Fit-for-purpose tools assists with subsystem monitoring and bottleneck analysis.
1.3 The System x Performance Lab IBM puts significant effort into ensuring that its servers have the highest performance level possible. Part of this effort is the System x Performance Lab, a group in Research Triangle Park, North Carolina, where work is done on System x servers through the development phase and after the servers become publicly available. During the development phase, the label creates performance models using subsystem and system functional specifications, chip functional specifications,
Chapter 1. Introduction to this IBM Redbook
5
input from the IBM development engineering departments, as well as trace information from the performance lab to do the following: Optimize the performance of the subsystem and system before the product is manufactured. Make design decision trade-offs. Select the optimum performance among various available chipsets that are intended to be used as part of the subsystem or system. Select optimum settings of the chipset parameters. This information is used to provide subsystem and system design guidance to the development engineering departments. As the system development phase nears completion, performance measurements are made with prototype subsystems and systems as well as with ship-level systems to do the following: Perform stress testing. Validate product functional specifications. Validate the subsystem and system performance models. Optimize the performance of the subsystem and system. Improve performance of third-party vendors tools, adapters, and software packages to perform well on System x servers. Develop performance white papers for marketing demonstrating the competitiveness of the System x systems. Develop performance tuning guides for customers using specified applications. Marketing and sales departments and vendors use this information to sell the System x systems, and customers can use this information to select the appropriate system and to tune their systems for their applications. To provide performance data, the System x Performance Lab uses the following benchmarks:
6
SAP Standard Application SD Benchmark 2-Tier / 3-Tier TPC-App Benchmark Application Server and Web Services TPC-C Benchmark (TPC-C) Transaction processing TPC-E Benchmark (TPC-E) Transaction processing TPC-H Benchmark (TPC-H) Ad hoc decision support TPC-DS Benchmark (TPC-DS) decision support Oracle 10g RAC Analysis SPECweb2005 (World Wide Web Server Content)
Tuning IBM System x Servers for Performance
SPEC CPU2000 Floating point and integer benchmarks SPECjbb2005 (Java™ Business Benchmark) MS Exchange MMB3 Mail Benchmark Terminal Services Benchmarks
1.4 IBM Center for Microsoft Technologies The IBM Center for Microsoft Technologies (CMT), located a few minutes from the Microsoft campus in Redmond, Washington, is the primary interface that IBM has with Microsoft in support of products that run on all IBM platforms. Positioned at CMT are highly trained IBM technical professionals who are dedicated to exploiting Windows XP, Windows 2000, Windows Server 2003, and SQL Server on Intel and AMD-based System x servers. The Center for Microsoft Technologies works in four areas: Development of device drivers, BIOS, Service Processor, Baseboard Management Controller, and Windows code for System x servers, including development of new technologies for the Windows platforms. Testing of IBM systems in the IBM Microsoft-Certified Hardware Compatibility Lab for both the Microsoft-designed hardware compatibility testing (HCT) and the more demanding Microsoft system compatibility testing (SCT). IBM applications being developed for Windows operating systems are also tested for Microsoft standards compliance here. Providing defect support with IBM Level 3 Support in high-severity situations when it is necessary to work directly with Microsoft Development personnel to resolve problems. The CMT also serves as a technical backup for the IBM Help Centers and as a worldwide center of IBM expertise in installation planning. Providing technical support for enterprise large accounts through the Windows Solutions Lab, which allows customers and independent software and hardware venders to run their workloads on System x servers.
1.5 Linux Technology Center The Linux Technology Center (LTC) serves as a center of technical competency for Linux both within IBM and externally. It provides technical guidance to internal software and hardware development teams and fulfills the role of an IBM extension to the open source Linux development community. The LTC is a worldwide development team within IBM whose goal is to use world-class programming resources and software technology from IBM to
Chapter 1. Introduction to this IBM Redbook
7
actively accelerate the growth of Linux as an enterprise operating system while simultaneously helping IBM brands exploit Linux for market growth. The LTC currently has programmers involved in many Linux projects, including scalability, serviceability, OS security, network security, networking, file systems, volume management, performance, directory services, standards, documentation, accessibility, test, security certification, systems management, cluster management, high availability, storage and I/O, PowerPC® support, power management, reliability, internationalization, and other projects that are required to make Linux a mature operating system that is ready for mission-critical workloads. Members of the LTC work directly in the open source community using standard open source development methodology. They work as peers within the shared vision of the Linux community leadership and participate in setting Linux design and development direction.
1.6 Understanding the organization of this IBM Redbook We have organized this IBM Redbook as follows: 1. 2. 3. 4. 5.
Understanding hardware subsystems Understanding operating system performance Working with performance monitoring tools Detecting and removing performance bottlenecks Tuning applications
After the introductory chapters, we have divided the chapters into parts to make it easier to find the information that you need: Part 2, “Server subsystems” on page 29 covers each of the major subsystems and their contributions to the overall performance of the server: – – – – – – –
CPU Chipsets PCI bus architecture Memory Disk subsystem Network adapter Operating system
Part 3, “Operating systems” on page 293 describes performance aspects of the operating systems that are covered in this book: – Windows Server 2003 and Windows 2000 Server – Red Hat Enterprise Linux and SUSE LINUX Enterprise Server – VMware ESX Server
8
Tuning IBM System x Servers for Performance
Part 4, “Monitoring tools” on page 469 covers the tools that are available to users of System x servers that run these operating systems, in addition to Capacity Manager, a component of IBM Director. With these tools, it is possible to capture and analyze performance data such that you can identify and remove both existing and future performance bottlenecks. Part 5, “Working with bottlenecks” on page 625 takes these tools to the next step by describing how to use them. It includes: – How to spot a performance problem and solve it quickly – A detailed explanation of the analysis of performance bottlenecks – Case studies that show real-life examples of performance analysis Part 6, “Applications” on page 743 describes the performance tuning approach for some of the popular server applications, such as: – – – – – – – –
Lotus Domino Microsoft Exchange SAP R/3® DB2 UDB Microsoft SQL Server Oracle Windows Terminal Services and Citrix Microsoft IIS
Chapter 1. Introduction to this IBM Redbook
9
10
Tuning IBM System x Servers for Performance
2
Chapter 2.
Understanding server types To optimize server performance, it is important to first understand the intended use of the system and the performance constraints that might be encountered. When you have identified the critical subsystems, you can then focus your attention on these components when resolving performance issues. This chapter describes the common server types and the subsystems that are most likely to be the source of a performance bottleneck. When defining the bottlenecks for server types, we list them in order of impact. This chapter discusses the following topics:
2.1, “Server scalability” on page 12 2.2, “Authentication services” on page 13 2.3, “File servers” on page 15 2.4, “Print servers” on page 16 2.5, “Database servers” on page 17 2.6, “E-mail servers” on page 18 2.7, “Web servers” on page 19 2.8, “Groupware servers” on page 20 2.9, “Multimedia server” on page 21 2.10, “Communication server” on page 21 2.11, “Terminal server” on page 22 2.12, “Infrastructure servers” on page 24 2.13, “Virtualization servers” on page 26 2.14, “High Performance Computing” on page 26
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
11
2.1 Server scalability Scalability is about increasing the capability of the server to allow the services that are provided to meet increased demands. Server scalability is generally achieved by adopting either scale-out or scale-up strategies, which are defined as follows: Scale-up is where server type subcomponents are increased in capacity to meet the increase in demand. For example, in a file server where the memory subsystem is a potential bottleneck, the amount of memory in the server can be increased to accommodate demand. For enterprise customers, the IBM System x3950 is a prime example of a server that is suited for scale-up. Scale-out is where multiple separate servers function as a single system. Scale-out is generally achieved through a form of load-balancing. For example, Microsoft Network Load Balancing offers scalability by balancing incoming client requests across clusters of individual servers. Tools such as NLB require you to install and configure additional components on the operating system. Thus, analyzing bottlenecks will become more complex. For enterprise customers, the IBM BladeCenter is a prime example of a server complex that is suited for scale-out. There are some server types that support applications that are capable of supporting their own scale-out options. Citrix and Weblogic are two examples. There are also solutions that work at the network layer called network load balancers. These are different from Microsoft Network Load Balancing because a device in the network controls incoming traffic and redirects network traffic to a number of individually grouped servers that provide a single service. For example, Radware Web Server Director is, in essence, a network device that will load balance incoming requests to a number of Web servers. Determining which approach to adopt influences how performance tuning is done. For example, while it is important to be able to identify potential bottlenecks, it is also important to understand how to resolve them. Attempting to add additional capacity to subcomponents that are at their maximum threshold will not resolve a bottleneck and the answer might be to scale out. Likewise, undertaking analysis of a server that is located inside a network load balanced-cluster will be more complex than troubleshooting a individual server.
12
Tuning IBM System x Servers for Performance
Table 2-1 lists server types and some of the scaling options that are available to medium to large customers. Table 2-1 Server types and scalability options Server type
Scale option
Scale method
File server
Scale out
Windows load balance
Print servers
Scale up
Hardware
Terminal servers
Scale out
Native
Web servers
Scale out
Network load balance
E-mail servers
Scale up
Hardware
Database servers
Scale up
Hardware
Computation servers
Scale out
Native
2.2 Authentication services Domain controllers provide authentication services and are central to the management of network resources including users, devices, and computers. They maintain and apply rules to provide secure and reliable working environments. Domain controllers communicate with each other continually to ensure that all rules are maintained consistently throughout the environment. For example, these servers communicate to ensure that user accounts, security settings, access control lists, and policies are synchronized. Domain controllers perform the following functions: User authentication Resource access validation Security control Common implementations are LDAP and Microsoft Active Directory®.
Chapter 2. Understanding server types
13
2.2.1 Windows Server 2003 R2 Active Directory domain controllers Windows Server 2003 R2 Active Directory domain controllers are the latest generation of the Windows domain controllers. Active Directory domain controllers provide improved server management, access management and storage management, and new 64-bit and .NET technologies. Some of these features are:
More efficient WAN replication Cross-platform Web single sign-on and password synchronization New management tools for file and printers File screening limits files types allowed
You can configure Windows Server 2003 R2 servers as domain controllers using the Active Directory wizard. You need to have an Active Directory in place to create additional domains. You can migrate Windows NT® 4.0 / 2000 / 2003 domains to the Windows 2003 R2 Active Directory, or you can access the existing domain through the Active Directory domain structure. Either of these scenarios can be helpful as part of your migration plan. Active Directory stores domain-wide directory data such as user authentication data, system security policies, and network objects (such as printer names and computer names) in its LDAP databank and provides the required tools to manage user and domain interactions, such as the logon process and validation, resource allocation, and directory searches. The Knowledge Consistency Checker (KCC) constructs and maintains the replication topology for Active Directory automatically, but administrators can also configure which domain members are synchronized with other domain members. Windows Server 2003 R2 Active Directory domains uses a new compression algorithm named Remote Differential Compression which improves the Distributed File System replication. The improved functionality that Windows 2003 R2 Active Directory delivers does have an impact on hardware performance and therefore the domain controllers might require additional memory and CPU cycles to service requests. Active Directory also provides a high level of integration with applications. For example, Microsoft Exchange Server 2003 uses Active Directory to store user information. Other server products are also likely to use Active Directory to store resource information in its LDAP databank. Because all Active Directory objects use fully qualified domain names (FQDNs), DNS is an extremely important service. If Active Directory is installed on a system that is running Windows Server 2003, an existing DNS service must be in place or a new DNS server implemented.
14
Tuning IBM System x Servers for Performance
Active Directory servicing requests quickly with the Windows Server 2003 domain servers requires adequate network bandwidth to perform synchronization, logon validation, and other services. In a domain controller, there are two kind of activities: Server-to-server activity These activities include the replication of the Active Directory partitions to the other Domain Controllers in your domain structure. There are five main partitions and a various number of application partitions. For example, one partition is the domain partition. In this partition are the computer, organizational units, and user objects that are stored. Client-to-server activity These activities include log on validation processes and LDAP requests. Windows Server 2003 R2 supports Kerberos authentication for high security encryption. The following hardware subsystems are sources of possible bottlenecks for Windows 2003 R2 domain controllers: Memory Network Processor
2.3 File servers The role of the file server is to store, retrieve, and update data that is dependent on client requests. Therefore, the critical areas that impact performance are the speed of the data transfer and the networking subsystems. The amount of memory that is available to resources such as network buffers and disk I/O caching also influence performance greatly. Processor speed or quantity typically has little impact on file server performance. In larger environments, you should also consider where the file servers are located within the networking environment. It is advisable to locate them on a high-speed backbone as close to the core switches as possible. The subsystems that have the most impact on file server performance are: Network Memory Disk
Chapter 2. Understanding server types
15
Tip: A common misconception is that CPU capacity is important. CPU is rarely a source of performance bottlenecks for file servers. The network subsystem, particularly the network interface card or the bandwidth of the LAN itself, might create a bottleneck due to heavy workload or latency. Insufficient memory can limit the ability to cache files and thus cause more disk activity, which results in performance degradation. When a client requests a file, the server must initially locate, then read and forward the requested data back to the client. The reverse of this applies when the client is updating a file. Therefore, the disk subsystem is potentially a bottleneck. See Chapter 22, “File and print servers” on page 745 for a discussion about how to improve the performance of file servers.
2.4 Print servers Print servers remove the requirement to install printers on individual clients and are capable of supporting a large number of printer types and print queues. They manage client print requests by spooling the print job to disk. The printer device itself can influence performance, because having to support slower printers with limited memory capacity takes longer to produce output while using resources on the Print Server. Therefore, the critical areas that impact performance are the speed of the data transfer and memory configuration. By default, the spool directory is located on the same disk as the operating system files, but it is better to redirect the directory to an other physical drive than the operating system disk. See Chapter 22, “File and print servers” on page 745 for a discussion about how to improve the performance of print servers. The subsystems that have the most impact on print server performance are: Memory Disk Processor Implementing printer pools and virtual printer configurations might help to reduce printing workload.
16
Tuning IBM System x Servers for Performance
2.5 Database servers The database server’s primary function is to store, search, retrieve, and update data from disk. Examples of Database engines include IBM DB2, Microsoft SQL Server, and Oracle. Due to the high number of random I/O requests that database servers are required to do and the computation intensive activities that occur, the potential areas that have the most impact on performance are:
Memory Disk Processor Network
The subsystems that have the most impact on database server performance are: Memory subsystem Buffer caches are one of the most important components in the server and both memory quantity and memory configuration is a critical factor. If the server does not have sufficient memory, paging occurs, which results in excessive disk I/O, which in turn generates latencies. Memory is required for both the operating system and the database engine. You need to consider this when sizing database servers. See the following sections to determine how much memory is needed: – Windows: 19.3, “Analyzing memory bottlenecks” on page 661 – Linux: 20.3, “Memory subsystem bottlenecks” on page 694 Disk subsystem Even with sufficient memory, most database servers will perform large amounts of disk I/O to bring data records into memory and flush modified data to disk. The disk sub-storage system needs to be well designed to ensure that it is not a potential bottleneck. Therefore, it is important to configure a sufficient number of disk drives to match the CPU processing power that is used. With most database applications, more drives equals greater performance. It is also important to keep your logs files on different disks to your database. Even when using SAN devices for storage, you need to pay particular attention to Fibre channel network and SAN configuration to ensure that the storage environment does not place constraints on the server.
Chapter 2. Understanding server types
17
CPU subsystem Processing power is another important factor for database servers because database queries and update operations require intensive CPU time. The database replication process also requires considerable amounts of CPU cycles. Database servers are multi-threaded applications. So, SMP-capable systems provide improved performance scaling to 16-way and beyond. L2 cache size is also important due to the high hit ratio—the proportion of memory requests that fill from the much faster cache instead of from memory. For example, SQL Server’s L2 cache hit ratio approaches 90%. Network subsystem The networking subsystem tends to be the least important component on an application or database server because the amount of data returned to the client is a small subset of the total database. The network can be important, however, if the application and the database are on separate servers. A balanced system is especially important, for example, if adding additional CPUs, consider upgrading other subsystems such as increasing memory and ensuring that disk resources are adequate. In database servers, the design of an application is critical (for example, database design, and index design).
2.6 E-mail servers E-mail servers act as repositories and routers of electronic mail, and they handle the transfer of e-mail to its destination. Because e-mail servers need to communicate regularly to do directory replication, mail synchronization, and interface to third-party servers, they do generate network traffic. Because they also have to store and manage mail, the disk subsystem is becoming increasingly more important. The important subsystems for e-mail servers are:
Memory CPU Disk Network
E-mail servers use memory to support database buffers and e-mail server services. Ensuring that memory is sized appropriately and that the disk subsystems are effective is very important because these will impact server
18
Tuning IBM System x Servers for Performance
performance. For example, if memory size is sufficient, the server is capable of caching more data, which results in improved performance. E-mail servers use log files to transfer modified data to an information store. These log files are written sequentially, which means that new transactions are appended to the end of the transaction files. Log files and database files have different usage patterns with log files performing better with separate physical disks, and database files performing better with striped disk arrays due to the random workload. Using several drives instead of a single drive can significantly increase e-mail throughput. Read-ahead disk-caching disk subsystems can also offer performance benefits. Users’ mailboxes can be stored either on the server or on each user’s local hard drive, or both. In each case, you need high network performance because clients still retrieve their mail over the network. The larger the size of the e-mails the more bandwidth that is required. Also, server-to-server replication traffic can be a significant load in the network and using multiple LAN adapters can help to improve its network performance. When an e-mail server receives a message it determines where the appropriate server is to handle the e-mail. If the address is local, it is stored in the database of the e-mail server. If the address is not local, the e-mail is forwarded on to the most appropriate server for processing. If the address is a distribution list, the server checks the addresses in the list and routes the message accordingly. These processes requires CPU cycles and sufficient memory must be allocated to ensure that these processes occur efficiently. If your server supports directory replication and connectors between sites, your server will experience high distribution list usage, and the CPU will be a more important factor in e-mail server performance. Adequate network bandwidth between e-mail servers and their clients is essential. However contrary to popular belief this is not the most impacted subsystem. If IPsec is to be used to encrypt network traffic, using a specialized network card to offload the encryption process will reduce CPU utilization.
2.7 Web servers Today, a Web server is responsible for hosting Web pages and running server-intensive Web applications. If Web site content is static, the subsystems that might be sources of bottlenecks are: Network Memory CPU
Chapter 2. Understanding server types
19
If the Web server is computation-intensive (such as with dynamically created pages), the subsystems that might be sources of bottlenecks are:
Memory Network CPU Disk
The performance of Web servers depends on the site content. There are sites that use dynamic content that connect to databases for transactions and queries and this requires additional CPU cycles. It is important that in this type of server there is adequate RAM for caching and to manage the processing of dynamic pages for a Web server. Also, additional RAM is required for the Web server service. The operating system automatically adjusts the size of cache depending on requirements. Because of high hit ratio and transferring large dynamic data, the network can be another potential bottleneck.
2.8 Groupware servers Groupware servers such as Lotus Notes and Microsoft Exchange, among others, are designed to allow user communities to share information and this enhances the teamwork concept for company users. It is usually implemented in a client/server model. Important subsystems include: Memory CPU Disk I/O Groupware servers generally support public folder access, scheduling, calendering, collaboration applications, and workflow applications. These systems require significant CPU power similar to e-mail servers. Routing and real-time collaboration require additional CPU cycles. Memory is used for caching just as it is for e-mail servers, and groupware servers use a special memory cache design to increase the data access rate. Therefore, the server should be configured with enough memory to eliminate or reduce paging to disk. Groupware servers are transactional-based client/server database applications and similar to database servers, the disk subsystem is an important factor in performance.
20
Tuning IBM System x Servers for Performance
When designing groupware systems, particular attention should be paid to amount of server-to-server traffic anticipated and slow LAN/WAN links must be considered.
2.9 Multimedia server Multimedia servers provide the tools and support to prepare and publish streaming multimedia presentations utilizing your intranet or the Internet. They require high-bandwidth networking and high-speed disk I/O because of the large data transfers. If you are streaming audio, the most probable sources of bottlenecks are: Network Memory Disk If you are streaming video, most important subsystems are: Network Disk I/O Memory Disk is more important than memory for a video server due to the volume of data being transmitting and the large amount of data being read. If the data is stored on the disk, disk speed is also an important factor in performance. If compression/decompression of the streaming data is required, then CPU speed and amount of memory are important factors as well.
2.10 Communication server Communication servers provides remote connection to your LAN and the most popular communication server is the Windows 2003 remote access services (RAS) server. A communication server’s bottlenecks are usually related to the speed of the communication lines and cards themselves. Typically, these applications do not put a stress on the processor, disk, or memory subsystems and the speed of the communication line will dictate the performance of the communication server. A high-speed T1 line, for example, causes less performance degradation than a 56 Kbps line.
Chapter 2. Understanding server types
21
The subsystems that are the most probable sources of bottlenecks are: Communication lines These are the physical connections between the client and server and as mentioned above the most critical performance factor is the speed of these communication lines. You should select faster communication lines to achieve better performance. Digital communications You should select digital lines if possible because they are more efficient at transferring data and transmit with fewer errors. Digital communications also benefit because fault detection and correction software and hardware might not have to be implemented. Port configuration Port is the input/output source for the communication devices. For example, if you have modem devices you should configure your port speed, flow control, and buffering to increase data flow performance. Other features, such as multilink and pure digital communications will help to improve performance. Correct configuration of the operating system’s port status and use of the correct device driver are other important steps in maintaining high performance.
2.11 Terminal server Windows Server 2003 Terminal Services enables a variety of desktops to access Windows applications through terminal emulation. In essence, the application is hosted and executed on the terminal server and only screen updates are forwarded to the client. It is important to first understand the factors in terminal server performance: Your application – – – –
Application memory requirements Shareable application memory Application screen refresh rate Applications typing requirements
Your users – – – – –
22
Typing speed Leave the applications open Logon time Logged on all day long or not Most logins at a specific time of the day or not
Tuning IBM System x Servers for Performance
Your network – – – –
Users’ typing speed Applications are graphic-intensive or not Client workstations’ display resolutions Application network bandwidth requirements
The subsystems that are the most probable sources of bottlenecks are: Memory CPU Network As the terminal servers execute applications and send the results to the client workstation, all the processing load is on the server. Terminal servers require powerful CPUs and sufficient memory. Because these servers can support multiple concurrent clients, the network is another important subsystem. Terminal servers do not benefit from large L2 cache sizes primarily because they have a very large working set. The working set is the number of instructions and data that are frequently accessed by the CPU. This working set is too large, and addresses generated by terminal server applications are more random across this large address space than most server applications. As a result, most terminal server configurations will obtain minimal benefits from large L2 processor caches. Generally, double the number of users requires double the performance of the CPU and double the amount of memory. CPU and memory requirements increase linearly, so you should use SMP-capable servers. The following factors also affect performance: Hard-disk throughput (for higher performance, use RAID devices) High-bandwidth network adapters Intelligent dial-up communications adapter (to reduce interrupt overhead and increase throughput)
Chapter 2. Understanding server types
23
2.12 Infrastructure servers Infrastructure servers is the name given to DNS, DHCP, WINS, and other services that provide connectivity.
2.12.1 DNS server Domain Name System (DNS) is a protocol for naming computers and network services. It is used to locate computers and services through user-friendly names. When a client uses a DNS name, DNS services can resolve the name to other information associated with that name, such as an IP address. The number of requests that the DNS server is required to respond to will be determined by the size of the environment that it is supporting and the number of DNS servers that will be located within that environment. You should consider these factors when sizing the server type. Important subsystems include: Network Memory The network subsystem, particularly the network interface card or the bandwidth of the LAN itself, can create a bottleneck due to heavy workload or latency. Insufficient memory might limit the ability to cache files and thus cause more disk and CPU activity, which results in performance degradation. Due to the nature of DNS serving, the processor subsystem is the least likely to cause a bottleneck.
2.12.2 DHCP server Dynamic Host Configuration Protocol (DHCP) is a protocol for using a server to manage and administer IP addresses and other related configuration items in the network. When a device starts, it might issue a request to obtain a IP address. The DHCP server responds and provides that device with a valid IP address that is valid for a predefined period of time. This protocol removes the requirement to assign individual IP addresses for each device. The number of requests that the DHCP server is required to respond to and the size of IP address scope will be critical in determining the server size. Having multiple DHCP and splitting the scope might reduce overheads on individual servers.
24
Tuning IBM System x Servers for Performance
Important subsystems include: Network Disk Memory The network subsystem, particularly the network interface card or the bandwidth of the LAN itself, can create a bottleneck due to heavy workload or latency. High disk I/O requests require an appropriately designed disk subsystem. Insufficient memory might limit the ability to cache files and thus cause more disk and CPU activity, which results in performance degradation. Due to the nature of DHCP serving, the processor subsystem is the least likely to cause a bottleneck.
2.12.3 WINS server Windows Internet Name Service (WINS) is a system that resolves NetBIOS names to IP addresses. For example, when a client uses a NetBIOS reference, the WINS server can resolve the NetBIOS name to other information associated with that name, such as an IP address. The number of requests that the WINS server is required to respond to will be determined by the size of the environment that it is supporting and the number of WINS servers that are located within that environment. You should consider these factors when sizing the server type. Important subsystems include: Network Disk Memory The network subsystem, particularly the network interface card or the bandwidth of the LAN itself, might create a bottleneck due to heavy workload or latency. High disk I/O requests require an appropriately designed disk subsystem. Insufficient memory might limit the ability to cache files and thus cause more disk and CPU activity, which results in performance degradation. Due to the nature of WINS serving, the processor subsystem is the least likely to cause a bottleneck.
Chapter 2. Understanding server types
25
2.13 Virtualization servers Virtualization servers provide the ability to run multiple simultaneous servers (or virtual machines on a single hardware platform. This is achieved by installing a product such as VMware ESX Server, which then provides the capability to divide the hardware subsystems into smaller partitions that then appear as multiple individual servers. These partitions can then be configured with an operating system and function as a traditional server type. For example, a server with two CPUs and 2 GB of RAM with 300 GB of disk can be partitioned into four servers, each with ½ CPU and 500 MB of RAM with 75 GB of disk. These servers could then be configured as different server types. For example, they can be configured as a Active Directory server, WINS server, DNS server, and DHCP server. The benefit is that servers that have spare capacity can be reconfigured as multiple different servers thereby reducing the number of physical servers that need to be supported in the environment. The individual virtual server type will still have the same potential bottlenecks and performance issues as the physical server type and there is still the added overhead of having to support the virtualization layer. Potential bottlenecks on the virtual operating system are: Memory CPU Network
2.14 High Performance Computing Computation servers provide floating-point and memory resources to compute-intensive applications such as are found in high-performance computing (HPC). These servers are often clustered together using extremely high-speed interconnects, such as Myrinet or InfiniBand, to provide significantly greater computational performance than would otherwise be available to a single server alone. Typical applications are characterized by their dependence on 32-bit or 64-bit floating-point operations. A computation server’s bottlenecks are generally related to the speed with which floating-point computations can be performed. Numerous factors can affect that, including the native vector or scalar performance of the processor and the size of processor cache. Vector operations are arithmetic operations that repeatedly
26
Tuning IBM System x Servers for Performance
perform the same operation on streams of related data. Scalar operations are working on each element separately. The speed at which data can be retrieved from memory is often one of the most important performance bottlenecks. Many HPC applications stride through large arrays in a uniform manner that brings data into the processor, uses it for a few operations, then writes a result back to memory. This characteristic is unfriendly to caches and pushes the performance bottleneck out to main memory. Computation servers need high network latency or throughput performance. To accomplish this, they are connected through high speed interconnects such as Myrinet, InfiniBand, or Quad. Depending on the specific HPC workload types, every technology has its own advantages. Potential bottlenecks on the virtual operating system are: Memory Network CPU
Chapter 2. Understanding server types
27
28
Tuning IBM System x Servers for Performance
Part 2
Part
2
Server subsystems In this part, we explain the technology that is implemented in the major subsystems in System x servers and show what settings you can make or adjust to obtain the best performance. We provide rules of thumb to give advice on what to expect from any changes that you consider. We examine closely each of the major subsystems so that you can find specific bottlenecks, and we present options as to what you can do to resolve these bottlenecks. We also discuss how to anticipate future bottlenecks. This part includes the following chapters:
Chapter 3, “Introduction to hardware technology” on page 31 Chapter 4, “Processors and cache subsystem” on page 35 Chapter 5, “Virtualization hardware assists” on page 79 Chapter 6, “PCI bus subsystem” on page 85 Chapter 7, “Chipset architecture” on page 97 Chapter 8, “Memory subsystem” on page 129 Chapter 9, “Disk subsystem” on page 169 Chapter 10, “Network subsystem” on page 235
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
29
30
Tuning IBM System x Servers for Performance
3
Chapter 3.
Introduction to hardware technology Servers are made up of a number of subsystems, each of which plays an important role in how the server performs. Depending on the use of the server, some of these subsystems are more important and more critical to performance than others. This chapter defines the server subsystems.
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
31
3.1 Server subsystems The subsystems in a server are: Processor and cache The processor is the heart of the server and is involved in most transactions that occur in the server. While the CPU is an important subsystem, the mistaken belief exists that the CPU is often the source of a performance bottleneck so buying a server with the fastest CPU is best. In the majority of server installations, the CPU is, in fact, over-powered and the other subsystems are under-powered. It is only specific applications that are truly CPU intensive that take advantage of the full power of today’s multi-core and 64-bit processors. The classic example of a server that does not need much CPU power is the file server (this is, by coincidence, the most common use of a server). Most file request traffic uses direct memory access (DMA) techniques to bypass the CPU and rely in the network, memory, and disk subsystems for throughput capacity. There are a number of processors available from Intel and AMD that are used in System x servers today. It is important to understand their differences and their strengths. Cache, while strictly part of the memory subsystem, is physically packaged with the processor these days. The CPU and cache are coupled together tightly and run at full or half the speed of the processor. In this book, we have grouped the cache and processor together. PCI bus The PCI bus is the pipe along which all data traverses into and out of the server. All System x servers use the PCI bus (PCI-X and PCI Express) for their critical adapter resources, SCSI and disk for example. High-end servers now have multiple PCI buses and many more PCI slots than they used to. Advances in the PCI bus include the PCI Express (PCI-E) 1X to 16X technologies, which provide greater throughput and connectivity options. Connecting to the CPU and cache is the PCI chipset. This set of components governs the connections between the PCI bus and the processor and memory subsystems. The PCI chipset is carefully matched and tuned to the processors and memory to ensure the maximum performance of the system. Memory Critical to a server’s performance is memory. Without enough memory installed, the system will perform poorly because the operating system will swap data to disk when it needs to make room for other data in memory.
32
Tuning IBM System x Servers for Performance
A feature in the Enterprise X-Architecture System x servers is memory mirroring for increased fault tolerance. The feature, part of IBM Active Memory™, is roughly equivalent to RAID-1 in disk arrays, in that memory is divided in two ports and one port is mirrored to the other. All mirroring activities are handled by the hardware without any additional support by the operating system. New memory technologies include the Fully Buffered DIMMs (FBD), providing higher capacity and bandwidth, improved flexibility and memory mirroring. Disk Perhaps the most configurable subsystem from an administrator’s point of view, the disk subsystem is often critical to a server’s performance. In the pyramid of online storage devices (cache, memory and disk), disk drives are by far the slowest and also the biggest, mainly because they are mechanical components. For many server applications, most data accessed will be stored on disk, so a fast disk subsystem is very important. To maximize capacity, RAID is commonly employed in server configurations. However, the configuration of the RAID arrays can make a significant difference in the performance characteristics. First, the choice of RAID level for the defined logical drives will affect performance as well as capacity and fault tolerance, which you would normally expect. There are many RAID levels available using the IBM ServeRAID and IBM Fibre Channel adapters and each has its place in specific server configurations. Equally important for performance reasons is the number of hard disks you configure in each array: the more disks, the better the throughput. An understanding of how RAID handles I/O requests is critical to maximizing performance. Serial technologies are now available on all System x servers to improve price-performance and scalability. These include Serial ATA (SATA) and Serial-attached SCSI (SAS). Network The network adapter card is the server’s interface to the rest of the world. If the amount of data through this portal is significant, then an under-powered network subsystem will have a serious impact on server performance. Beyond the server, the design of your network is equally important. The use of switches to segment the network or the use of such technologies as ATM should be considered. 1 Gbps network adapters are now commonplace in servers. New 10 Gbps network are now available to provide the necessary bandwidth for high-throughput applications. Moreover, new technologies such as TCP Offload Engine help improve performance.
Chapter 3. Introduction to hardware technology
33
Video The video subsystem in a server is relatively insignificant. The only use of it is when an administrator is working on the server’s console. Production users will never make use of the video, so emphasis is rarely placed on this subsystem. We do not cover video in this book for this reason. Operating system We consider the operating system to be a subsystem that can be the source of bottlenecks similar to other hardware subsystems. Windows, Linux, and ESX Server have settings that you can change to improve performance of the server. This part of the book describes each of the major subsystems in detail. It explains how they are important and what you can do to tune them to your requirements. The subsystems that are critical to performance depend on what you are using the server for. The bottlenecks that occur can be resolved by gathering and analyzing performance data; however, this task is not a one-time job. Bottlenecks can vary depending on the workload coming into the server and can change day-to-day and week-to-week.
34
Tuning IBM System x Servers for Performance
4
Chapter 4.
Processors and cache subsystem The central processing unit (CPU or processor) is the key component of any computer system. In this chapter, we cover several different CPU architectures from Intel (IA32, Intel 64 Technology, IA64) and AMD1 (AMD64) and outline their main performance characteristics. This chapter discusses:
4.1, “Processor technology” on page 36 4.2, “64-bit computing” on page 61 4.3, “Processor performance” on page 66 4.4, “Rules of thumb” on page 76 Note: In this book, we collectively refer to the processors from Intel (IA32, Intel 64 Technology) and AMD (AMD64) as Intel compatible processor.
Intel 64 Technology is the new name for Extended Memory 64 Technology (EM64T).
1
Content in this chapter is reprinted by permission of Advanced Micro Devices, Inc.
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
35
4.1 Processor technology The central processing unit has actually outperformed all other computer subsystems in its evolution. Thus, most of the time, other subsystems such as disk or memory will impose a bottleneck upon your application (unless pure number crunching or complex application processing is the desired task). Understanding the functioning of a processor in itself is already quite a difficult task, but today IT professionals are faced with multiple and often very different CPU architectures. Comparing different processors is no longer a matter of looking at the CPU clock rate but one of understanding what CPU architecture is best suited for what kind of workload. Also, 64-bit computing has finally made its move from the high-end UNIX® and mainframe systems into the Intel compatible arena and has become yet another new technology to be understood. The Intel compatible microprocessor has evolved from the first 8004 4-bit CPU, produced in 1971, to the current line of Xeon® and Core processors. AMD, on the other hand, has stepped out of Intel’s shadow with the world’s first IA32 compatible 64-bit processor. Our overview of processors begins with the current line of Intel Xeon CPUs, followed by the AMD Opteron and Intel’s Core Architecture. For the sake of simplicity, we will not explore earlier processors.
4.1.1 Single core Intel Xeon processors The server version of the Pentium® 4 processor is named the Intel Xeon processor and exists in either a version for dual socket systems, the Xeon DP, and a multi-processor variant, the Xeon MP. Without additional logic, the Xeon MP natively supports up to a four-socket SMP configuration. With the IBM XA-64e chipset, configurations can be up to 32-sockets. The Pentium 4 (P4) introduced the first significant micro-architecture change for the processor since the introduction of the Pentium Pro P6 core in 1995 (P6 core was used in the Pentium Pro through to the Pentium III Xeon processor). This new P4 architecture used in the Xeon DP and Xeon MP processors is named NetBurst. The new Xeon Processor uses the ZIF socket design instead of the Slot 2 cartridge design of the Pentium III Xeon processors. This smaller form factor allows for up to eight processors to be installed in a 4U server such as the xSeries® 445.
NetBurst architecture Under the name NetBurst, Intel introduced the first significant architectural change since the P6 architecture introduced by the Pentium Pro. The NetBurst architecture is primarily aimed at very high clock rates and explains Intel’s
36
Tuning IBM System x Servers for Performance
leadership in CPU clock rates. The main feature of the NetBurst architecture are as follows: 20 stage execution pipeline The older Pentium III Xeon processor has a 10-stage pipeline. However, the large number of transistors in each pipeline stage meant that the processor was limited to speeds of about 1.3 GHz due to greater latency in the pipeline. The Xeon processor has a 20-stage pipeline (the Xeon processors with Intel 64 Technology have an even longer 31-stage pipeline), which can process up to 126 concurrent instructions in flight and up to 48 reads and 24 writes active in the pipeline. The lower complexity of each stage enable NetBurst processors to scale up to a clock rate of approximately 5 GHz. Such an increase of the CPU clock would however come at the price of even greater power consumption. As a result, Intel will not continue long-term development based on the NetBurst architecture but rather is moving toward more power-efficient processor microarchitecture as described on “Intel Core microarchitecture” on page 50. Cache architecture Compared to the Pentium III class of processors, the Xeon line of CPUs features a drastically increased level one (L1) cache. The first Xeon processors code named Foster feature a L1 data cache of 8 KB and an instruction cache (Execution Trace Cache) of 12 KB. It should, however, be pointed out that this large L1 cache is required because the longer pipeline architecture used in the new P4 processors could experience longer stall times during cache miss operations. So the larger caches are important to keep the very long pipeline of the NetBurst architecture from stalling. Double clock arithmetic Sometimes also referred to as Rapid Execution Engine, this feature means that simple integer operations are processed at twice the actual processor clock speed. Within the processor, the two arithmetic logical units (ALU) can execute most operations within half of a clock cycle. (This is why integer operations are processed at twice the CPU clock.) However, this feature does not lead to twice the integer performance because instructions cannot go through the pipeline faster than the actual CPU clock. The purpose of this feature, just as with the new cache architecture, is to keep the very long pipeline from stalling. That said, we should not underestimate the significant integer performance of a Xeon processor. Compact floating-point Basic floating-point operations are processed faster thanks to this feature of the NetBurst architecture. Also, the registers for floating-point operations are expanded to a width of 128 bits plus additional registers for the status of data movement.
Chapter 4. Processors and cache subsystem
37
Streaming 128-bit arithmetic SSE2 or Streaming SIMD Execution 2 gives the Xeon processor the ability to perform multiple integer or floating-point operations at a time. SIMD refers to Single Instruction Multiple Data which effectively gives the processor the ability to perform an operation such as ADD to multiple data sets. Of course, the basic requirement for SIMD to work is having enough data sets at hand that can be processed using the very same operation. SIMD or SSE is especially handy when it comes to multimedia applications such as graphics processing or MPEG rendering. However, vectorized scientific problems such as matrix multiplications can also benefit from the SSE2 engine. Accelerated system bus The Pentium III Xeon processor has a 100 MHz front-side bus that equates to a peak burst throughput of 800 MBps. Servers that execute protocols such as TCP/IP often generate bottlenecks on the front-side bus due to multiple buffer copies that are required by the protocol. TCP/IP protocol can result in the contents of each data packet travelling as much as five times over the front-side bus. This data packet travelling has been shown to be a bottleneck in high-throughput situations. The Xeon processor improves this situation by using two 100 MHz clocks, out of phase with each other by 90°, and by using both edges of each clock to transmit data, as shown in Figure 4-1.
100 MHz clock A 100 MHz clock B Figure 4-1 Quad-pumped front-side bus
This configuration increases the performance of the 100 MHz front-side bus by a factor of four without the difficulty of high-speed clock signal integrity issues. The end result is an effective burst rate of 400 MHz and with 8 bytes of data transfer per clock the effective throughput is 3.2 GBps. This faster front-side bus can have a substantial impact on performance, especially for TCP/IP intensive LAN traffic. More recent versions of the Xeon processor have 677 MHz (166 MHz quad pumped), and 800 MHz (200 MHz quad pumped) front-side bus speeds. These more recent processors provide performance improvements in performance for any high-memory bandwidth applications such as intensive TCP/IP transfers, image and video streaming. For more information about the NetBurst architecture, see: http://www.intel.com/cd/ids/developer/asmo-na/eng/44004.htm
38
Tuning IBM System x Servers for Performance
Hyper-Threading Hyper-Threading technology effectively enables a single physical processor to execute two separate code streams (threads) concurrently. To the operating system, a processor with Hyper-Threading appears as two logical processors, each of which has its own architectural state—that is, its own data, segment, and control registers and its own advanced programmable interrupt controller (APIC). Each logical processor can be individually halted, interrupted, or directed to execute a specified thread, independently of the other logical processor on the chip. However, unlike a traditional 2-way SMP configuration that uses two separate physical processors, the logical processors share the execution resources of the processor core, which include the execution engine, the caches, the system bus interface, and firmware. Figure 4-2 illustrates the basic layout of a Hyper-Threading-enabled CPU. As shown in the figure, only the components for the architectural state of the CPU have doubled. Hyper-Threading technology is designed to improve server performance by exploiting the multi-threading capability of operating systems, such as Windows Server 2003 and Linux, and server applications, in such a way as to increase the use of the on-chip execution resources that are available on these processors. Physical processor Architectural state
Cache
Processor without Hyper-Threading
Processing resources
Logical processor Architectural state
Architectural state
Cache
Processing resources
Processor with Hyper-Threading
Figure 4-2 Hyper-Threading processor versus a non-Hyper-Threading processor
Having fewer or slower processors usually yields the best gains in performance when comparing Hyper-Threading on versus Hyper-Threading off because with fewer processors there is a greater likelihood that the software can spawn sufficient numbers of threads to keep both paths busy. The performance gains from Hyper-Threading running on slower speed processors are usually greater
Chapter 4. Processors and cache subsystem
39
than the gains that are obtained when running on high-speed processors because on the slower processors there are longer periods of time between serialization points that nearly every software must use. Whenever two threads must serialize, performance is reduced. The performance gains that are obtained in a highly parallel threaded database environment from enabling Hyper-Threading are as follows: Two physical processors: up to about 35% performance gain Four physical processors: up to about 33% performance gain Eight physical processors: up to about 30% performance gain Over time, these gains in performance will change because software developers will introduce improved threading, which makes more efficient use of Hyper-Threading. However, much of the currently available software often limits SMP scalability, but we can expect improved results as software matures. Best-case multi-threaded applications today are:
Databases SAP Web servers PeopleSoft® VMware 64-bit Terminal Services
Table 4-1 on page 41 lists the support for Hyper-Threading for various operating systems. The table uses the following terms: None indicates that the operating system does not recognize the logical processors that Hyper-Threading enables. Yes indicates that the operating system recognizes the logical processors and can execute threads on them but is not optimized for Hyper-Threading. From a licensing perspective, if Hyper-Threading is enabled, the operating system will need to be licensed for twice the number of physical processors to take full advantage of the processors’ capabilities. Optimized indicates that the operating system recognizes the logical processors and the operating system code has been designed to take full advantage of the technology. From a licensing perspective, the logical processors do not count towards the number of processors for which an operating system is licensed. Ignores indicates that the operating system recognizes the logical processors when Hyper-Threading is enabled but that the operating system ignores them.
40
Tuning IBM System x Servers for Performance
Table 4-1 Operating system support for Hyper-Threading Description
Hyper-Threading
Windows NT Enterprise Edition
None
Windows 2000 Server
Yesa
Windows 2000 Advanced Server
Yesa
Windows 2000 Datacenter Server
Yesa
Windows Server 2003, Standard Edition
Optimized
Windows Server 2003, Enterprise Edition
Optimized
Windows Server 2003, Datacenter Edition
Optimized
Linux kernel 2.4.18+
Yes
Linux kernel 2.6
Optimized
VMware ESX Server 2.x
Yes
VMware ESX Server 3
Optimized
a. Microsoft supports Hyper-Threading as described in the document “Microsoft Windows-Based Servers and Intel Hyper-Threading Technology,” available from http://www.microsoft.com/windows2000/docs/hyperthreading.doc. However, the Intel BIOS Programmers Guide recommends disabling Hyper-Threading on systems that are running Windows 2000 Server or earlier or Linux kernels earlier than 2.4.18.
Linux kernels from 2.4.18 and later and Windows 2000 SP3 and Windows Server 2003 operating systems understand the concept of physical processors versus logical processors. However, there are no plans from Microsoft to patch Windows NT 4.0 Enterprise Edition. Important: Fully optimized scheduler support for Hyper-Threading is available with the 2.6 series of Linux kernels. For more information about Hyper-Threading, see: http://www.intel.com/technology/hyperthread/
Chapter 4. Processors and cache subsystem
41
Single-core Intel Xeon DP specifications The Intel Xeon DP processor is designed as a powerful CPU for high-end workstation and servers that scale up to two physical CPUs. To date, there have been five iterations of the Xeon DP. Xeon DP Foster The first Intel Xeon DP processor (code named Foster) is available at clock frequencies of 1.40, 1.50, 1.70, and 2 GHz. Unlike later versions of the Intel Xeon DP and MP, Foster did not feature Hyper-Threading technology. With the exception of Hyper-Threading, the Foster processor supports the whole feature set of the NetBurst architecture, however. Other features of Foster Xeon DP include: – A 400 MHz front-side bus – A level-1 cache that is divided into 12 KB of instruction cache and 8 KB of data cache with a 4-way associativity – 256 KB of advanced transfer L2 cache that is 8-way associative – New SSE2 instructions Xeon DP Prestonia The successor to Foster, Prestonia, was the first Xeon DP processor to support Hyper-Threading. Other differences between Prestonia and Foster include double the L2 cache size L2 cache and the manufacturing process (0.13 micron versus 0.18 micron used for the Foster). – Clock frequencies of 1.80, 2, 2.20, 2.40, 2.60, and 2.80 GHz – Hyper-Threading support – 512 KB of advanced transfer L2 cache with 8-way cache associativity and ECC Xeon DP Prestonia (refresh) The next iteration of the Xeon DP processor, also code named Prestonia, includes upgrades in clock frequency, front-side bus, and new L3 cache. The differences with the first Xeon DP (Foster) are as follows: – Clock frequencies of 2, 2.40, 2.66, 3.06, and 3.20 GHz – Hyper-Threading support – 533 MHz front-side bus with a capacity of 4.3 GBps – 512 KB of advanced transfer L2 cache with 8-way cache associativity and ECC – 1 MB and 2 MB (available with 3.20 GHz versions) L3 cache with 8-way associativity and ECC
42
Tuning IBM System x Servers for Performance
Xeon DP Nocona The follow-on to Prestonia of the Xeon DP processor series is code named
Nocona. In fact, the features that the Nocona introduces are so numerous that it can also be regarded as a new processor. It is built using 90 nm process technology and includes Intel 64 Technology (EM64T) instruction set, which effectively expands the 32-bit instruction set used in the IA32 to 64 bits. Due to Intel 64 Technology, Nocona supports a 36-bit physical address space. With 36-bit physical addressing, the Nocona processor is capable of addressing up to 64 GB of real memory. This is four times greater than the memory that can be installed into any current dual-processor system which in general is limited to eight DIMM slots. The major differences of the Nocona compared to the first Xeon DP processor are: – – – – – – – –
Intel 64 Technology (EM64T) support with 36-bit physical addressing Clock frequencies of 2.80, 3, 3.20, 3.40, and 3.60 GHz Hyper-Threading support Front side bus speed of 800 MHz A 16 KB L1 cache 1 MB of advance transfer L2 cache 13 new SSE3 instructions Additional general purpose registers and additional floating-point registers
Xeon DP Irwindale The Irwindale processor allows for a updated Nocona core with the addition of a larger L2 cache of 2 MB instead of 1 MB and the ability to reduce power consumption when the processor was not heavily used. In addition, a frequency of 3.8 GHz is offered with Irwindale. Intel 64 Technology is discussed in “64-bit extensions: AMD64 and Intel 64 Technology” on page 62 and 64-bit technology is discussed in detail in 4.2, “64-bit computing” on page 61. For more information about the Xeon DP, see: http://www.intel.com/design/xeon/prodbref
Single-core Intel Xeon MP The Xeon DP processor is designed to be used in high-end workstations and two-socket servers. For high-end multi-processing environments, Intel offers the Xeon MP.
Chapter 4. Processors and cache subsystem
43
Because Intel designed the Xeon MP to work in SMP configurations where the front-side bus could be a possible bottleneck, the MP processors generally features up to three levels of cache, all of which are on the same processor die (refer to 7.2.2, “SMP” on page 101 for additional information about SMP). There have been four iterations of the single core Xeon MP processors: Xeon MP Foster The first Intel Xeon MP processor (code named Foster MP) is available at clock frequencies of 1.40, 1.50, and 1.6 GHz. Unlike the Foster DP processor, Foster MP did include Hyper-Threading technology. The Foster processors support the entire feature set of the NetBurst architecture. Other features of Foster Xeon MP include: – A 400 MHz front-side bus – A level-1 cache that is divided into 12 KB of instruction cache and 8 KB of data cache with a 4-way associativity – 256 KB of advanced transfer L2 cache that is 8-way associative – A L3 cache of 512 KB or 1 MB – New SSE2 instructions Xeon MP Gallatin The Gallatin processor is the MP version of the Prestonia DP processor. The Gallatin processor performed much faster then Foster MP and featured a larger L3 cache. Other differences between Gallatin and Foster MP include doubling the L2 cache and the manufacturing process (0.13 micron versus 0.18 micron used for the Foster MP). – Clock frequencies of 1.5, 1.9, 2.0, 2.2, 2.5, 2.7, 2.8, and 3.0 GHz – Hyper-Threading support – 512 KB of advanced transfer L2 cache with 8-way cache associativity and ECC – L3 cache size of 1 MB, 2 MB, or 4 MB Xeon MP Potomac Potomac processors are the most recent single core Xeon processors. They were built to support Intel 64 Technology and similarly to the other MP Xeon processor, they support three levels of cache on the processor: – L3 cache size of 4 MB or 8 MB. This cache features 4-way or 8-way associativity. – Level 2 cache size of 1 MB. The L2 cache implements the Advanced Transfer Cache technology, which means L2-to-processor transfers occur
44
Tuning IBM System x Servers for Performance
across a 256-bit bus in only one clock cycle. Also the L2 cache is implemented as an 8-way associative cache with ECC. – Similar to the other Intel MP Xeon processors, an L1 cache, 12 KB in size, is closest to the processor and is used to store micro-operations (that is, decoded executable machine instructions) and serves those to the processor at rated speed. This additional level of cache saves decode time on cache hits. There is an additional 8 KB for data related to those instructions. As well as the additional cache level (which is also standard in later Intel Xeon DP versions), there is also support for Hyper-Threading and the NetBurst characteristics discussed in “Single-core Intel Xeon DP specifications” on page 42. Xeon MP Cranford The addition of a large L3 cache could increase latencies for some workloads. Server type workloads with highly random access patterns might experience many L3 cache misses which would increase the latency of the overall system. For that reason, a single core MP processor without a large L3 cache was created, code named Cranford. Cranford processors run at higher speeds of 3.17 GHz and 3.67 GHz and have a 1 MB L2 cache. They are identical to Potomac processors with the exception that they do not have the large L3 cache. For more information about the features of the Xeon Processor MP, go to: http://www.intel.com/design/xeon/xeonmp/prodbref
4.1.2 Dual core Intel Xeon processors Moore’s Law states that the number of transistors on a chip doubles about every two years. Similarly, as the transistors have become smaller, the frequency of the processors have increased which is generally equated with performance. However, around 2003, physics started to limit advances in obtainable clock frequency. Transistors sizes have become so small that electron leakage through transistors has started to occur. Those electron leaks result in large power consumption and substantial extra heat and could even result in data loss. In addition, the ability to cool the processor at higher frequencies has become too expensive with the traditional air cooling methods. This is why the material that comprises the dielectric in the transistors has become a major limiting factor in the frequencies that are obtainable. Manufacturing advances have continued to enable a higher per-die transistor count but have only been able to obtain about a 10% frequency improvement per year. For that reason, processor vendors are now placing more processors on
Chapter 4. Processors and cache subsystem
45
the die to offset the inability to increase frequency. Multi-core processors provide the ability to increase performance with lower power consumption. Intel released their first dual core Xeon processors in October 2005. Dual core processors are two separate physical processors combined onto a single processor socket. Dual core processors consist of twice as many cores but each core is run at lower clock speeds as an equivalent single core chip in order to lower the waste heat usage. Waste heat is the heat that is produced from electron leaks in the transistors. There are five dual core Xeon processor models available in IBM System x servers: Xeon DP and Xeon MP Paxville The Paxville DP was the first dual core Xeon processor released. This processor has a frequency of 2.8 MHz, 800 MHz front-side bus, and two 2 MB L2 caches. This processor essentially consists of two single core “Irwindale” processors combined into one core. The Paxville DP processor is Intel’s first generation dual core processor. Paxville MP (Intel Xeon 7000 series) processors incorporate all of the main features/design that are provided in the single core Irwindale processor. In addition, Paxville MP processors include the Virtualization Technology to allow for a single machine to run multiple operating systems simultaneously. See Chapter 5, “Virtualization hardware assists” on page 79 for information. The following features are included with the Paxville MP processor: – – – – – – – – – –
Netburst microarchitecture Intel 64 Technology support with 36-bit physical addressing Frequencies of 2.67, 2.8, and 3.0 GHz Hyper-Threading Technology support Front-side bus speed of 667 MHz or 800 MHz A 16 KB L1 cache 1 MB or 2 MB of advance transfer L2 cache SSE3 instructions Additional general purpose registers and additional floating-point registers Virtualization Technology support
Note: The Paxville DP dual-core Intel Xeon processor has an architectural limitation such that if you disable Hyper-Threading in BIOS, then the second core is also disabled. This limitation only applies to the Paxville DP processor which is installed in servers such as the x336 and x346. See RETAIN® tip H187141 for details: http://www.pc.ibm.com/support?page=MIGR-65101
46
Tuning IBM System x Servers for Performance
Xeon 5000 Series DP processor (Dempsey) Using 65nm technology and the Netburst microarchitecture, Dempsey is a Xeon DP processor which is the successor to the Paxville DP series. Dempsey allows for faster frequencies of up to 3.73 GHz and front-side bus speeds up to 1066 MHz. In addition, all features from the Paxville MP series are incorporated into this processor. Dempsey processors are supported in the Intel 5000 series chipsets, as described in 7.3.3, “Intel 5000 chipset family” on page 114. The Dempsey processors consume between 95 W and 130 W, significantly higher than the follow-on Woodcrest as described below. Xeon 7100 Series MP processor (Tulsa) Tulsa follows the Xeon MP line and is the follow on to the Paxville MP processor. Tulsa is similar in architecture to Paxville with the main exception that it includes a shared L3 cache and is built on a 65 nm technology. Previous dual core Intel processors did not include an L3 cache. The key advantages of the Tulsa processor is: – Frequencies of 2.5, 2.6, 3.0, 3.2, 3.3, and 3.4 GHz which provides a greater selection of processor frequencies then the Paxville MP processor. – 1 MB L2 cache in each core and a shared L3 cache ranging in size from 4-16 MB. The L3 cache is shared instead of a separate one in each core. – Front-side buses of 677 and 800 MHz. Figure 4-3 on page 48 illustrates the overall structure of the Tulsa processor. The Tulsa processor includes two Dempsey cores each with a 1 MB L2 cache. L3 cache is much faster then the previous Potomac processor’s L3 cache. The Potomac processor’s L3 cache experienced high latencies in determination of a cache miss which is improved greatly in the Tulsa L3 cache. One major issue that occurs with early Intel dual core processors is that the cores within the same processor socket are unable to communicate directly to each other internally. Instead, cores within the same processor use the external front-side bus to transmit data between their individual caches. Tulsa processors incorporate a shared L3 cache between cores. Because both cores share the same L3 cache, core-to-core data communication can occur internally within the processor instead of externally on the front-side bus. Traffic that occurred previously on the front-side bus is now moved internally into the processor, which frees up front-side bus traffic.
Chapter 4. Processors and cache subsystem
47
Dempsey core 0
Dempsey core 1
1 MB L2 cache
1 MB L2 cache Cache-to-cache data sharing performed through the shared L3 cache
Up to 16 MB of L3 cache
Bus interface
Figure 4-3 Tulsa processor architecture
Xeon 5100 Series DP processor (Woodcrest) The Woodcrest processor is the first Xeon DP processor that uses the Intel Core microarchitecture instead of the Netburst microarchitecture. See 4.1.4, “Intel Core microarchitecture” for details. Frequencies of 1.6-3.0 GHz are supported with an L2 cache of 4 MB. The front-side bus runs at a frequency of either 1066 or 1333 MHz as shown in Table 4-2. None of these processors support Hyper-Threading. Woodcrest uses a low power model incorporated in the Core microarchitecture. This is an improvement of the 95W-130W power consumption of its predecessor, Dempsey. In addition to the substantial performance-per-watt increases, the Core microarchitecture of the Woodcrest processor provides substantial improvements in random memory throughput applications. Table 4-2 Woodcrest processor models
48
Processor model
Speed
L2 cache
Front-side bus
Power (TDP)
Xeon 5110
1.6 GHz
4 MB
1066 MHz
65 W
Xeon 5120
1.86 GHz
4 MB
1066 MHz
65 W
Xeon 5130
2.00 GHz
4 MB
1333 MHz
65 W
Tuning IBM System x Servers for Performance
Processor model
Speed
L2 cache
Front-side bus
Power (TDP)
Xeon 5140
2.33 GHz
4 MB
1333 MHz
65 W
Xeon 5148 LV
2.33 GHz
4 MB
1333 MHz
40 W
Xeon 5150
2.66 GHz
4 MB
1333 MHz
65 W
Xeon 5160
3.0 GHz
4 MB
1333 MHz
80 W
4.1.3 Quad core Intel Xeon processors Quad-core processors differ from single-core and dual-core processors by providing four independent execution cores. While some execution resources are shared, each logical processor has its own architecture state with its own set of general-purpose registers and control registers to provide increased system responsiveness. Each core runs at the same clock speed. Intel Quad code processors include the following: Xeon 5300 Series DP processor (Clovertown) The Clovertown processor is a quad-core design that is actually made up of two Woodcrest dies in a single package. Each Woodcrest die has 4 MB of L2 cache so the total L2 cache in Clovertown is 8 MB. The Clovertown processors are also based on the Intel Core microarchitecture as described in 4.1.4, “Intel Core microarchitecture”. Processor models available include the E5310, E5320, E5335, E5345 and E5355. The processor front-side bus operates at either 1066 MHz (processor models ending in 0) or 1333 MHz (processor models ending in 5). For specifics, see Table 4-3. None of these processors support Hyper-Threading. In addition to the features of the Intel Core microarchitecture, the features of the Clovertown processor include: – Intel Virtualization Technology — processor hardware enhancements that support software-based virtualization. – Intel 64 Architecture (EM64T) — support for both 64-bit and 32-bit applications. – Demand-Based Switching (DBS) — technology that enabled hardware and software power management features to lower average power consumption of the processor while maintaining application performance. – Intel I/O Acceleration Technology (I/OAT) — reduces processor bottlenecks by offloading network-related work from the processor.
Chapter 4. Processors and cache subsystem
49
Table 4-3 Clovertown processor models Processor model
Speed
L2 cache
Front-side bus
Power (TDP)
DemandBased Switching
E5310
1.6 GHz
8 MB
1066 MHz
80 W
No
E5320
1.86 GHz
8 MB
1066 MHz
80 W
Yes
E5335
2.0 GHz
8 MB
1333 MHz
80 W
No
E5345
2.33 GHz
8 MB
1333 MHz
80 W
Yes
E5355
2.66 GHz
8 MB
1333 MHz
120 W
Yes
Tigerton Tigerton will be the first Intel Xeon MP quad core processor and will be based on the Intel Core microarchitecture. It will be the follow-on to the Tulsa processor. Tigerton is due to be available in late 2007.
4.1.4 Intel Core microarchitecture The new Intel Core microarchitecture is based on a combination of the energy efficient Pentium M microarchitecture found in mobile computers as well as the current Netburst microarchitecture which is the basis for the majority of the Xeon server processors. The Woodcrest processor is the first processor to implement the Core microarchitecture. The key features of the Core microarchitecture include: Intel Wide Dynamic Execution The Core microarchitecture is able to fetch, decode, queue, execute, and retire up to four instructions simultaneously in the pipeline. The previous Netburst Microarchitecture was only able to run three instructions simultaneously in the pipeline. The throughput is improved effectively by processing more instruction in the same amount of time. In addition, certain individual instructions are able to be combined into a single instructions in a technique known as macrofusion. By combining instructions together, more instructions are able to fit within the pipeline. In order to use the greater pipeline throughput, the processors have more accurate branch prediction technologies and larger buffers so that there is a lower possibility of pipeline stalls and more efficient use of the processor.
50
Tuning IBM System x Servers for Performance
Intel Intelligent Power Capability The advantage of the Pentium M microarchitecture is the ability to use less power and enable a longer battery life of mobile computers. Similar technology has been improved and modified and added into the Core microarchitecture for high end computer servers. The Intelligent Power Capability provides fine-grained power control that enables sections of the processor that aren’t in use to be powered down. Additional logic is included in components such as ALU unit, FP unit, cache logic, and bus logic that improves power consumption on almost an instruction-by-instruction basis. Processor components can then be powered on the instant it is needed to process an instruction, with minimal lag time so that performance is not jeopardized. Most importantly, the actual power utilization is substantially lower with the Core microarchitecture due to this additional power control capability. Intel Advanced Smart Cache The L2 cache in the Core microarchitecture is shared between cores instead of each core using a separate L2. Figure 4-4 illustrates the difference in the cache between the traditional Xeon with Netburst microarchitecture and the Intel Core microarchitecture.
Netburst dual core (not Tulsa) CPU 0
CPU 1
2 MB L2 cache
2 MB L2 cache
Intel Core Architecture
CPU 0
CPU 1
4 MB shared cache
Bus interface
Bus interface
Cache-to-cache data sharing performed through the shared L2 cache (faster)
Cache-to-cache data sharing performed through the bus interface (slower)
Figure 4-4 Intel Xeon versus Intel Core Architecture
The front-side bus utilization would be lower, similar to the Tulsa L3 shared cache as discussed in “Xeon 7100 Series MP processor (Tulsa)” on page 47. With a dual-core processor, the Core microarchitecture allows for one, single
Chapter 4. Processors and cache subsystem
51
core to use the entire shared L2 cache if the second core is powered down for power saving purposes. As the second core begins to ramp up and use memory, it will allocate the L2 memory away from the first CPU until it reaches a balanced state where there is equal use between cores. Single core performance benchmarks, such as SPEC Int and SPEC FP, benefit from this architecture because single core applications are able to allocate and use the entire L2 cache. SPEC Rate benchmarks balance the traffic between the cores and more effectively balance the L2 caches. Intel Smart Memory Access Intel Smart Memory Access allows for additional technology in the processor to prefetch more often, which can assist performance. Previously, with the NetBurst architecture, when a write operation appeared in the pipeline, all subsequent read operations would stall until the write operation completed. In that case, prefetching would halt, waiting for the write operation to complete, and the pipeline would fill with nop or stall commands instead of productive commands. Instead of making forward progress, the processor would make no progress until the write operation completes. By Intel’s memory disambiguation technologies, additional load-to-memory operations are executed prior to a store operation completing. If the load instruction turns out to be incorrect, the processor is able to back out the load instruction and all dependent instructions that might have executed based on that load. However, if the load instruction turns out to be valid, the processor has spent less time waiting and more time executing, which improves the overall instruction level parallelism. Intel Advanced Digital Media Boost Advanced Digital Media Boost increased the execution of SSE instructions from 64 bits to 128 bits. SSE instructions are Streaming Single Instruction Multiple Data Instructions that characterize large blocks of graphics or high bandwidth data applications. Previously, these Instructions would need to be broken into two 64-bit chunks in the execution stage of the processor but now support has been included to have those 128-bit instructions executed at one per clock cycle. For more information about the Core microarchitecture, go to: http://www.intel.com/technology/architecture/coremicro
4.1.5 Opteron processors The Opteron processor, code named Sledgehammer, was introduced in April 2003 after a major effort by AMD to deliver the first 64-bit processor capable of running in 64-bit mode and running existing 32-bit software as fast as or faster than current 32-bit Intel processors. Opteron was designed from the start to be
52
Tuning IBM System x Servers for Performance
an Intel-compatible processor, but AMD also wanted to introduce advanced technology that would set Opteron apart from processors developed by Intel. The Opteron processor has a physical address limit of 40-bit addressing (meaning that up to 1 TB of memory can be addressed) and the ability to be configured in 2-socket and 4-socket multi-processor configurations. Clearly, no current 2-socket or 4-socket can be configured with 1 TB of memory, but the 1 TB limit does indicate good future growth potential before a hardware change to the processor is needed. Apart from improved scalability, there is another significant feature introduced with the Opteron CPU, namely the 64-bit addressing capability of the processor. With the Itanium® processor family, Intel introduced a completely new 64-bit architecture that was incompatible with existing software. AMD decided there was significant market opportunity for running existing software and upgraded the existing x86 architecture to 64-bit. As with the transitions the x86 architecture underwent in the years before both the move to 16-bit with the 8086 and the move to 32-bit with the Pentium architecture, the new AMD64 architecture simply expanded the IA32 architecture by adding support for new 64-bit addressing, registers and instructions. The advantage of this approach is the ability to move to 64-bit without having to perform a major rewrite of all operating system and application software. The Opteron CPU has three distinct operation modes that enable the CPU to run in either a 32-bit mode or a 64-bit mode, or a mixture of both. This feature gives users the ability to transition smoothly to 64-bit computing. The AMD64 architecture is discussed in more detail in “64-bit extensions: AMD64 and Intel 64 Technology” on page 62. The Opteron CPU exists in various models that differ in their ability to scale up: AMD Opteron 100 series: single socket-processor support AMD Opteron 200 series: up to two socket processor support AMD Opteron 800 series: up to eight socket processor support
AMD Rev E (940 socket) Opteron specifications The common features of AMD Revision E 940-pin socket processors are as follows: CPU clocks from 1.6 to 3.0 GHz for single core models and 1.6 to 2.6 GHz for dual core models AMD64 which provides 32-bit or 64-bit computing on the same system 16 64-bit wide general purpose registers (GPRs) 16 64-bit wide SSE/SSE2 registers
Chapter 4. Processors and cache subsystem
53
Three HyperTransport links at 800 MHz or 1000 MHz with a 16-bit width – On the 100 series processors, HT links can only connect to I/O devices – On the 200 series processors, HT links can connect to I/O and one link can connect to another CPU (but only one link — the CPU is limited to 2-way processing). – On the 800 series processors, all HT links can connect either to I/O or to another CPU 64 KB of L1 data cache, 2-way associative with ECC protection 64 KB of L1 instruction cache, 2-way associative with parity protection 1 MB of L2 cache, 4-way associative with ECC protection
AMD Rev F (1207 socket) Opteron specifications AMD Revision F processors are the successor to the Rev E processors. They are not compatible with current Rev E processors but should be pin compatible with future quad core processors. The following features are incorporated into the Rev F processors: All rev F processors are multi-core. There will be no single core Opteron processors after the current generation. Support for DDR2 memory is added. The following number of memory DIMM slots are incorporated into the system: – Eight DIMMs at DDR2 speeds of 400 or 533 MHz instead of 266/333 MHz DDR1. – Four DIMMs at 667 MHz DDR2 instead of 400 MHz DDR1 – Two DIMMs eventually will be supported at 800 MHz DDR2 instead of 400 MHz DDR1 The current 1 GHz HyperTransport technology is incorporated as the interface to initial Rev F processors. PCI Express support is added. AMD Pacifica virtualization technology and power management technologies are incorporated. The major performance increase with Rev F AMD Opteron over the current Rev E AMD Opteron processors is related to the increase to DDR2 memory technology. DDR2 memory technology results in over a 25% increase in memory throughput for random memory accesses. Memory performance of DDR verses DDR2 is analyzed in detail in 8.1.5, “Double Data Rate memory, DDR and DDR2” on page 135.
54
Tuning IBM System x Servers for Performance
Figure 4-5 shows the architecture of the Opteron Revison E and Revision F processors.
Core 0
Core 1
1 MB L2 cache
1 MB L2 cache
System Request Interface Crossbar Switch
Memory controller
HT 1
HT 2
HT 3
Figure 4-5 Architecture of dual core Opteron Rev E and Rev F processors
Internally, each core of the processor is connected directly to a 1 MB L2 cache. Memory transfers between the two caches on the processors occur directly through a crossbar switch which is on the chip. By direct communication between the L2 caches on different cores, the HyperTransport is not used for processor core-to-core communication. The bandwidth of the HyperTransport can then be used to communicate with other devices or processors. The chipset that interfaces with the AMD Opteron processors as well as the HyperTransport links are discussed in more detail in 7.4, “PCI bridge-based chipsets” on page 123.
Chapter 4. Processors and cache subsystem
55
IBM CPU passthru card The AMD Opteron-based systems such as the System x3755 support up to four processors and their associated memory. In a four-way configuration, the x3755 has the configuration that is illustrated in Figure 4-6. Card 4
Card 1
DIMMs
DIMMs
DIMMs
DIMMs
Opteron processor
Opteron processor
PCI-E Bridge
PCI-E Bridge
DIMMs
Opteron processor Card 3
HTX slot
DIMMs
PCI Express x16 PCI Express x8
PCI Express x8 PCI Express x4
DIMMs
Gigabit Ethernet
DIMMs
Gigabit Ethernet SAS
Opteron processor
South Bridge
Card 2
Figure 4-6 Block diagram of the x3755 with four processors installed
Note that each adjacent processor is connected through HyperTransport links, forming a square. The third HT link on each processor card connects to an I/O device, or is unused.
56
Tuning IBM System x Servers for Performance
The x3755 also supports a 3-way configuration (by removing the processor card in slot 4, the upper-left quadrant as shown in Figure 4-6. This configuration results in processor connections as shown in the top half of Figure 4-7.
Without a passthru card, cards 1 & 3 are two hops away
Card 3
Card 2
Card 1
Passthru With a passthru card, cards 1 & 3 are connected
Card 3
Card 2
Card 1
Figure 4-7 The benefit of the passthru card for three-way configurations
However, with the addition of the IBM CPU Passthru card, part number 40K7547, the processors on cards 1 and 3 are directly connected together, as shown in the bottom half of Figure 4-7.
Chapter 4. Processors and cache subsystem
57
The passthru card basically connects two of the HyperTransport connectors together and provides a seamless connection between the processors on either side. The resulting block diagram of the x3755 is shown in Figure 4-8. Card 4
Card 1 DIMMs DIMMs
Passthru card
Opteron processor
PCI-E Bridge
PCI-E Bridge
DIMMs
Opteron processor Card 3
PCI Express x8
PCI Express x8 PCI Express x4
DIMMs HTX slot
DIMMs
PCI Express x16
Gigabit Ethernet DIMMs
Gigabit Ethernet SAS
Opteron processor Card 2
South Bridge
Figure 4-8 Block diagram of a three-way x3755 with a passthru card installed
There are performance benefits to adding the passthru card in a three-way configuration. Without the passthru card, the configuration requires that snoop requests and responses originating from one of the two end processors (see Figure 4-7 on page 57), and certain non-local references, travel over two hops. With the passthru card, this now becomes only one hop.
58
Tuning IBM System x Servers for Performance
The benefit in decreased latency and increased memory throughput as shown in Figure 4-9. Stream benchmark results 20000 Memory throughput (MBps)
18000 16000 14000 12000 10000 8000 6000 4000 2000 0 Copy
Scale
Add
Triad
Memory operation 3-way with Passthru card
3-way without Passthru card
Figure 4-9 Memory throughput benefit of the passthru card
For more information see the white paper Performance of the IBM System x 3755 by Douglas M Pase and Matthew A Eckl, available from http://www.ibm.com/servers/eserver/xseries/benchmarks/related.html
4.1.6 Itanium 2 processors The Itanium 2 was the first processor of the IA64 family for broad commercial use. The first Itanium was more or less a technology demonstration and was not deployed widely, largely because it did not achieve its intended performance targets and software portability severely limited the number of applications that were supported with IA64. Itanium 2 processors are often referred to as the IPF processors, where IPF stands for Itanium Processor Family. The processor uses a completely different
Chapter 4. Processors and cache subsystem
59
instruction set, the Itanium Instruction Set Architecture and is also referred to as Intel Itanium Architecture or IA64. Itanium 2 is based on the Explicitly Parallel instruction Computing (EPIC) instruction set (as opposed to a CISC or RISC design). With EPIC, three instructions are bundled together to form a single 128-bit structure. This bundling allows a fixed length instruction that is similar to the RISC architecture. EPIC attempts to improve performance by moving much of the out-of-order processing complexity into the software compiler. The result is a simpler CPU design and therefore higher productivity. The important point is that the compilers must be designed well to achieve a competitive result. An optimized compiler is much more important when using an IA64 compiler then on a IA32 or x64 processor. Itanium 2 is a so-called wide CPU architecture (in comparison to the deep architecture used in the Xeon family). This architecture means that the Itanium 2 can process more instructions per clock cycle than a current Intel Xeon processor, but because of the deeper pipelined architecture, the Xeon processors can run at faster clock speeds. The very wide architecture of the Itanium 2 comes however at a price. In order to keep the multiple pipelines of the Itanium 2 saturated, most software needs to be recompiled and in most cases redesigned to achieve the desired level of parallelism. The Itanium 2 processor architecture combines instructions and operands into fixed size words of 128-bit width (Figure 4-10). This architecture is sometimes also referred to as VLIW for Very Large Instruction Word. 128 bits (big endian)
127
87 86
Instruction 3 (41 bits)
46 45
Instruction 2 (41 bits)
5 4
Instruction 1 (41 bits)
0
Pointer (5 bits)
Figure 4-10 The Very Large Instruction Word architecture
In addition, significantly large caches have to be used to always have enough instructions at hand to keep the pipelines full. Cache memory always means more transistors on the processor die, fewer yield from the wafers and hence significantly bigger production costs exist.
60
Tuning IBM System x Servers for Performance
Itanium 2 and IA32 applications The EPIC architecture and the IA64 instruction set are incompatible with the instruction set used in the IA32 (in fact IA64 is closer to a RISC processor than it is to IA32) and hence IA32 software cannot run natively under IA64. In order to support the vast number of applications that are written for IA32, the Itanium 2 features a hardware-based IA32 emulator that is similar to the one used in the first Itanium processor. This way of emulating an IA32 CPU, however, is not nearly as fast as running the same code on a current Intel Xeon or AMD Opteron processor. Intel has, therefore, introduced a software-based IA32 translator called the IA-32 Execution Layer (IA-32 EL). This translator is part of modern operating systems such as Windows 2003 or current Linux distributions. Even with the aid of the IA-32 Execution Layer, you should only expect at most half the performance of current Intel Xeon or AMD Opteron processors. When the Itanium 2 is given optimized IA64 code to process, it is a very fast processor. While the Itanium 2 will most probably not outperform a Intel Xeon or AMD Opteron processor in running integer-intensive workloads, it is comparable to an Intel 64 Technology or AMD64 processor at floating-point calculations. Performance of Itanium as well as Intel 64 Technology and AMD64 are discussed more in 4.2, “64-bit computing” on page 61.
4.2 64-bit computing As discussed in 4.1, “Processor technology” on page 36, there are three 64-bit implementations in the Intel-compatible processor marketplace: Intel IA64, as implemented on the Itanium 2 processor Intel Intel 64 Technology, as implemented on the 64-bit Xeon DP and Xeon MP processors AMD AMD64, as implemented on the Opteron processor There exists some uncertainty as to the definition of a 64-bit processor and, even more importantly, the benefit of 64-bit computing. Definition of 64-bit: A 64-bit processor is a processor that is able to address 64 bits of virtual address space. A 64-bit processor can store data in 64-bit format and perform arithmetic operations on 64-bit operands. In addition, a 64-bit processor has general purpose registers (GPRs) and arithmetic logical units (ALUs) that are 64 bits wide.
Chapter 4. Processors and cache subsystem
61
The Itanium 2 has both 64-bit addressability and GPRs and 64-bit ALUs. So, it is by definition a 64-bit processor. Intel 64 Technology extends the IA32 instruction set to support 64-bit instructions and addressing, but are Intel 64 Technology and AMD64 processors real 64-bit chips? The answer is yes. Where these processors operate in 64-bit mode, the addresses are 64-bit, the GPRs are 64 bits wide, and the ALUs are able to process data in 64-bit chunks. Therefore, these processors are full-fledged, 64-bit processors in this mode. Note that while IA64, Intel 64 Technology, and AMD64 are all 64-bit, they are not compatible for the following reasons: Intel 64 Technology and AMD64 are, with exception of a few instructions such as 3DNOW, binary compatible with each other. Applications written and compiled for one will usually run at full speed on the other. IA64 uses a completely different instruction set to the other two. 64-bit applications written for the Itanium 2 will not run on the Intel 64 Technology or AMD64 processors, and vice versa.
64-bit extensions: AMD64 and Intel 64 Technology Both AMD's AMD64 and Intel 64 Technology (formerly known as EM64T) architectures extend the well-established IA32 instruction set with:
A set of new 64-bit general purpose registers (GPR) 64-bit instruction pointers The ability to process data in 64-bit chunks Up to 1TB of address space that physical memory is able to access 64-bit integer support and 64-bit flat virtual address space
Even though the names of these extensions suggest that the improvements are simply in memory addressability, both the AMD64 and the Intel Intel 64 Technology are in fact fully functional 64-bit processors. There are three distinct operation modes available in AMD64 and Intel 64 Technology: 32-bit legacy mode The first and, in the near future, probably most widely used mode is the 32-bit legacy mode. In this mode, both AMD64 and Intel 64 Technology processors will act just like any other IA32 compatible processor. You can install your 32-bit OS on such a system and run 32-bit applications, but you will not be able to make use of the new features such as the flat memory addressing above 4 GB or the additional General Purpose Registers (GPRs). 32-bit applications will run just as fast as they would on any current 32-bit processor.
62
Tuning IBM System x Servers for Performance
Most of the time, IA32 applications will run even faster because there are numerous other improvements that boost performance regardless of the maximum address size. For applications that share large amounts of data there might be performance impacts related to the NUMA-like architecture of multi-processor Opteron configurations since remote memory access might slow your application down. Compatibility mode The second mode supported by the AMD64 and Intel 64 Technology is compatibility mode which is an intermediate mode of the full 64-bit mode described below. In order to run in compatibility mode, you will need to install a 64-bit operating system and 64-bit drivers. If a 64-bit OS and drivers are installed both Opteron and Xeon processors will be enabled to support a 64-bit operating system with both 32-bit applications or 64-bit applications. Compatibility mode gives you the ability to run a 64-bit operating system while still being able to run unmodified 32-bit applications. Each 32-bit application will still be limited to a maximum of 4 GB of physical memory. However the 4 GB limit is now imposed on a per-process level, not at a system-wide level. This means that every 32-bit process on this system gets its very own 4 GB of physical memory space (assuming sufficient physical memory is installed). This is already a huge improvement compared to IA32 where the operating system kernel and the application had to share 4 GB of physical memory. Additionally, compatibility mode does not support the virtual 8086 mode, so real-mode legacy applications are not supported. 16-bit protected mode applications are however supported. Full 64-bit mode (Long Mode) The final mode is the full 64-bit mode. AMD refer to this as long mode and Intel refer to it as IA-32e mode. This mode is when a 64-bit operating system and 64-bit application are use. In the full 64-bit operating mode, an application can have a virtual address space of up to 40-bits (which equates to 1 TB of addressable memory). The amount of physical memory will be determined by how many DIMM slots the server has and the maximum DIMM capacity supported and available at the time. Applications that run in full 64-bit mode will get access to the full physical memory range (depending on the operating system) and will also get access to the new GPRs as well as to the expanded GPRs. However it is important to understand that this mode of operation requires not only a 64-bit operating system (and of course 64-bit drivers) but also requires a 64-bit application that has been recompiled to take full advantage of the various enhancements of the 64-bit addressing architecture. For more information about the AMD64 architecture, see: http://www.x86-64.org/
Chapter 4. Processors and cache subsystem
63
For more information about Intel 64 Technology, see: http://www.intel.com/technology/64bitextensions/
The benefit of 64-bit (AMT64, Intel 64 Technology) computing In the same way that 16-bit processors and 16-bit applications are no longer used in this space, it is likely that at some point in the future, 64-bit processors and applications will fully replace their 32-bit counterparts. Processors using the Intel 64 Technology and AMD64 architectures are making this transition very smooth by offering 32-bit and 64-bit modes. This means that the hardware support for 64-bit will be in place before you upgrade or replace your software applications with 64-bit versions. IBM System x already has many models available with the Intel 64 Technology-based Xeon and AMD64 Opteron processors. The question you should be asking is whether the benefit of 64-bit processing is worth the effort of upgrading or replacing your 32-bit software applications. The answer is that it depends on the application. Here are examples of applications that will benefit from 64-bit computing: Encryption applications Most encryption algorithms are based on very large integers and would benefit greatly from the use of 64-bit GPRs and ALUs. While modern high-level languages allow you to specify integers above the 232 limit, in a 32-bit system, this is achieved by using two 32-bit operands, thereby causing a significant overhead while moving those operands through the CPU pipelines. A 64-bit processor will allow you to perform 64-bit integer operation with one instruction. Scientific applications Scientific applications are another example of workloads that need 64-bit data operations. Floating-point operations do not benefit from the larger integer size because floating-point registers are already 80 or 128 bits wide even in 32-bit processors. Software Applications requiring more than 4 GB of memory The biggest advantage of 64-bit computing for commercial applications is the flat, potentially massive, address space. 32-bit enterprise applications such as databases are currently implementing Page Addressing Extensions (PAE) and Addressing Windows Extensions (AWE) addressing schemes to access memory above the 4 GB limit imposed by 32-bit address limited processors. With Intel 64 Technology and AMD64, these 32-bit addressing extension schemes support access to memory up to 128 GB in size.
64
Tuning IBM System x Servers for Performance
One constraint with PAE and AWE, however, is that memory above 4 GB can only be used to store data. It cannot be used to store or execute code. So, these addressing schemes only make sense for applications such as databases, where large data caches are needed. In contrast, a 64-bit virtual address space provides for direct access to up to 2 Exabytes (EB), and even though we call these processors 64-bit, none of the current 64-bit processors actually supports full 64 bits of physical memory addressing, simply because this is such an enormous amount of memory. In addition, 32-bit applications might also get a performance boost from a 64-bit Intel 64 Technology or AMD64 system running a 64-bit operating system. When the processor runs in Compatibility mode, every process has its own 4 GB memory space, not the 2 GB or 3 GB memory space each gets on a 32-bit platform. This is already a huge improvement compared to IA32 where the OS and the application had to share those 4 GB of memory. When the application is designed to take advantage of more memory, the availability of the additional 1 or 2 GB of physical memory can create a significant performance improvement. Not all applications take advantage of the global memory available. APIs in code need to be used to recognize the availability of more than 2 GB of memory. Furthermore, some applications will not benefit at all from 64-bit computing and might even experience degraded performance. If an application does not require greater memory capacity or does not perform high-precision integer or floating-point operations, then 64-bit will not provide any improvement. In fact, because 64-bit computing generally requires instructions and some data to be stored as 64-bit objects, these objects consume more physical memory than the same object in a 32-bit operating environment. The memory capacity inflation of 64-bit can only be offset by an application taking advantage of the capabilities of 64-bit (greater addressing or increased calculation performance for high-precision operations), but when an application does not make use of the 64-bit operating environment features, it often experiences the overhead without the benefit. In this case, the overhead is increased memory consumption, leaving less physical memory for operating system buffers and caches. The resulting reduction in effective memory can decrease performance. Software driver support in general is lacking for 64-bit operating systems compared to the 32-bit counterparts. General software drivers such as disk controllers or network adapters or application tools might not have 64-bit code in place for x64 operating systems. Prior to moving to an x64 environment it might
Chapter 4. Processors and cache subsystem
65
be wise to ensure that all third-party vendors and software tools support drivers for the specific 64-bit operating system that you are planning to use.
64-bit memory addressing The width of a memory address dictates how much memory the processor can address. A 32-bit processor can address up to 232 bytes or 4 GB. A 64-bit processor can theoretically address up to 264 bytes or 16 Exabytes (or 16777216 Terabytes), although current implementations address a smaller limit, as shown in Table 4-4. Note: These values are the limits imposed by the processors. Memory addressing can be limited further by the chipset implemented in the server. For example, the XA-64e chipset used in the x3950 Xeon based server addresses up to 512 GB of memory. Table 4-4 Memory supported by processors Processor
Flat addressing
Addressing with PAE
Intel 32-bit Xeon MP (32-bit) processors including Foster MP and Gallatin
4 GB (32-bit)
128 GB
Intel 64-bit Xeon DP Nocona (64-bit)
64 GB (36-bit)
128 GB in compatibility mode
Intel 64-bit Xeon MP Cranford (64-bit)
64 GB (36-bit)
128 GB in compatibility mode
Intel 64-bit Xeon MP Potomac (64-bit)
1 TB (40-bit)
128 GB in compatibility mode
Intel 64-bit (64-bit) Dual Core MP including Paxville, Woodcrest, and Tulsa
1 TB (40-bit)
128 GB in compatibility mode
AMD Opteron (64-bit)
256 TB (48-bit)
128 GB in compatibility mode
The 64-bit extensions in the processor architectures Intel 64 Technology and AMD64 provide a better performance for both 32-bit and 64-bit applications on the same system. These architectures are based on 64-bit extensions to the industry-standard x86 instruction set and provide support for existing 32-bit applications.
4.3 Processor performance Processor performance is a complex topic because the effective CPU performance is affected by system architecture, operating system, application, and workload. This is even more so with the choice of three different CPU architectures, IA32, IA64, and AMD64/EM64T.
66
Tuning IBM System x Servers for Performance
In general, server CPUs execute workloads that have very random address characteristics. This is expected because most servers perform many unrelated functions for many different users. So, core clock speed and L1 cache attributes have a lesser effect on processor performance compared to desktop environments. This is because with many concurrently executing threads that cannot fit into the L1 and L2 caches, the processor core is constantly waiting for L3 cache or memory for data and instructions to execute.
4.3.1 Comparing CPU architectures Every CPU we have discussed so far had similar attributes. Every CPU has two or more pipelines, an internal clock speed, L1 cache, L2 cache (some also L3 cache). The various caches are organized in different ways, some of them 2-way associative, some go up to 16-way associativity. Some have a 800 MHz FSB while others have no FSB at all (Opteron). Which is fastest? Is the Xeon DP the fastest CPU of them all because it is clocked at up to 3.6 GHz? Or is the Itanium 2 the fastest because it features up to 9 MB of L3 cache? Or is it perhaps the Opteron because its L2 cache features a 16-way associativity? As is so often the case, there is never one simple answer. When comparing processors, clock frequency is only comparable when comparing processors of the same architectural family. You should never compare isolated processor subsystems across different CPU architectures and think you can make a simple performance statement. Comparing different CPU architectures is therefore a very difficult task and has to take into account available application and operating system support. As a result, we do not compare different CPU architectures in this section, but we do compare the features of the different models of one CPU architecture.
4.3.2 Cache associativity Cache associativity is necessary to reduce the lookup time to find any memory address stored in the cache. The purpose of the cache is to provide fast lookup for the CPU, because if the cache controller had to search the entire memory for each address, the lookup would be slow and performance would suffer. To provide fast lookup, some compromises must be made with respect to how data can be stored in the cache. Obviously, the entire amount of memory would be unable to fit into the cache because the cache size is only a small fraction of the overall memory size (see 4.3.3, “Cache size” on page 71). The methodology of how the physical memory is mapped to the smaller cache is known as set associativity (or just associativity).
Chapter 4. Processors and cache subsystem
67
First, some definitions. Looking at Figure 4-11, main memory is divided up into pages. Cache is also divided up into pages and a memory pages is the same size as a cache page. Pages are divided up into lines or cache lines. Generally cache lines are 64 bytes wide. For each page in memory or in cache, the first line is labeled cache line 1, the second line is labeled cache line 2, and so on. When data in memory is to be copied to cache, the line that this data is in is copied to the equivalent slot in cache. Looking at Figure 4-11, when copying cache line 1 from memory page 0 to cache, it is stored in cache line 1 in the cache. This is the only slot it can be stored in cache. This is a one-way associative cache, because for any given cache line in memory, there is only one position in cache where it can be stored. This is also known as direct mapped, because the data can only go into one place in the cache. Main Memory cache line 1 cache line 2
page 0
One-way associative (Direct Mapped) L2 cache way-a cache line 1 cache line 2
cache line 1 cache line 2
cache line 1 cache line 2
Figure 4-11 One-way associative (direct mapped) cache
68
Tuning IBM System x Servers for Performance
page 1
page n
With a one-way associative cache, if cache line 1 in another memory page needs to be copied to cache, it too can only be stored in cache line 1 in cache. You can see from this that you would get a greater cache hit rate if you use greater associativity. Figure 4-12 shows the 2-way set associative cache implementation. Here there are two locations in which to store the first cache line for any memory page. As the figure illustrates, main memory on the right hand side will be able to store up to two cache line 1 entries concurrently. Cache line 1 for page 0 of main memory could be located in way-a of the cache while cache line 1 for page n of main memory could be located in way-b of the cache simultaneously. Main Memory cache line 1 cache line 2
page 0
Two-way associative L2 cache way-a
way-b
cache line 1 cache line 2
cache line 1 cache line 2
cache line 1 cache line 2
cache line 1 cache line 2
page 1
page n
Figure 4-12 A 2-way set associative cache
Chapter 4. Processors and cache subsystem
69
Expanding on a one-way and two-way set associative cache, a 3-way set associative cache (Figure 4-13) provides three location, a 4-way set associative cache provides four locations and an 8-way set associative cache provides eight possible locations in which to store the first cache line from up to eight different memory pages. Main Memory cache line 1 cache line 2
page 0
Three-way associative L2 cache
way-a cache line 1 cache line 2
way-b cache line 1 cache line 2
way-c cache line 1 cache line 2
cache line 1 cache line 2
cache line 1 cache line 2
page 1
page n
Figure 4-13 3-way set associative cache
Set associativity greatly minimizes the cache address decode logic necessary to locate a memory address in the cache. The cache controller simply uses the requested address to generate a pointer into the correct cache page. A hit occurs when the requested address matches the address stored in one of the fixed number of cache location associated with that address. If the particular address is not there, a cache miss occurs. Notice that as the associativity increases, the lookup time to find an address within the cache could also increase because more pages of cache must be searched. To avoid longer cache lookup times as associativity increases, the lookups are performed in parallel, however, as the associativity increases, so does the complexity and cost of the cache controller.
70
Tuning IBM System x Servers for Performance
For the high performance X3 Architecture systems such as the System x3950, lab measurements determined that the most optimal configuration for cache was 9-way set associativity, taking into account performance, complexity and cost. A fully associative cache in which any memory cache line could be stored in any cache location could be implemented, but this is almost never done because of the expensive (in both cost and die areas) parallel lookup circuits required. Large servers generally have random memory access patterns as opposed to sequential memory access patters. Higher associativity favors random memory workloads due to it’s ability to cache more distributed locations of memory.
4.3.3 Cache size Faster, larger caches usually result in improved processor performance for server workloads. Performance gains obtained from larger caches increase as the number of processors within the server increase. When a single CPU is installed in a four-socket SMP server, there is little competition for memory access. Consequently, when a CPU has a cache miss, memory can respond, and with the deep pipeline architecture of modern processors, the memory subsystem usually responds before the CPU stalls. This allows one processor to run fast almost independently of the cache hit rate. On the other hand, if there are four processors installed in the same server, each queuing multiple requests for memory access, the time to access memory is greatly increased, increasing the potential for one or more CPUs to stall. In this case, a fast L2 hit saves a significant amount of time and greatly improves processor performance. As a rule, the greater the number of processors in a server, the more gain from a large L2 cache. In general: With two CPUs, expect 4% to 6% improvement when you double the cache size With four CPUs, expect 8% to 10% improvement when you double the cache size With eight or more CPUs, you might expect as much as 10% to 15% performance gain when you double processor cache size Of course, there are diminishing returns as the size of the cache improves; these are simply rules of thumb for the maximum expected performance gain.
Chapter 4. Processors and cache subsystem
71
4.3.4 CPU clock speed Processor clock speed affects CPU performance because it is the speed at which the CPU executes instructions. Measured system performance improvements because of an increase in clock speed are usually not directly proportional to the clock speed increase. For example, when comparing a 3.0 GHz CPU to an older 1.6 GHz CPU, you should not expect to see 87% improvement. In most cases, performance improvement from a clock speed increase will be about 30% to 50% of the percentage increase in clock speed. So for the example above, you could expect about 26% to 44% system performance improvement when upgrading a 1.6 GHz CPU to a 3.0 GHz CPU.
4.3.5 Scaling versus the number of processor cores In general, the performance gains shown in Figure 4-14 can be obtained by adding CPUs when the server application is capable of efficiently utilizing additional processors, and of course, there are no other bottlenecks occurring in the system.
Relative Database Performance Scaling Relative Performance Improvement
Random Transaction Processing Workload 8 7 6.3
6
1.5x
5 4
1.5x
3 2 1
1.6x 1.7x
4.2
2.8
1.74 1
0 1 CPU 4 GB
2 CPU 8 GB
4 CPU 16 GB
8 CPU 32 GB
16 CPU 64 GB
Figure 4-14 Typical performance gains when adding processors
These scaling factors can be used to approximate the achievable performance gains that can be obtained when adding CPUs and memory to a scalable Intel IA-32 server. For example, begin with a 1-way 3.0 GHz Xeon MP processor and add another Xeon MP processor. Server throughput performance will improve up to about 1.7 times. Increase the number of Xeon processors to four and server performance
72
Tuning IBM System x Servers for Performance
can improve to almost three times greater throughput than the single processor configuration. At eight processors, the system has a bit over four times greater throughput than the single processor configuration and finally at 16 processors the performance increases to over six fold greater throughput than the single CPU configuration. High performing chipsets such as the XA-64e generally are designed to provide higher scalability than the average chipset. Figure 4-15 shows the performance gain of a high performing chipset such as X3 Hurricane chipset in the x3850 as processors are added assuming no other bottlenecks occurring in the system. Performance gains of 1.9 and 1.8 are possible in certain business workloads.
System x3850 Performance Scaling Relative Performance Improvement
Random Transaction Processing Workload 5 4 3.56
3 2 1
1.8x 1.9x
1.96
1
0 1 CPU 16 GB
2 CPU 32 GB
4 CPU 64 GB
Figure 4-15 System x3850 Performance Scaling when adding processors
Database applications such as IBM DB2, Oracle, and Microsoft SQL Server usually provide the greatest performance improvement with increasing numbers of CPUs. These applications have been painstakingly optimized to take advantage of multiple CPUs. This effort has been driven by the database vendors’ desire to post #1 transaction processing benchmark scores. High-profile industry-standard benchmarks do not exist for many applications, so the motivation to obtain optimal scalability has not been as great. As a result, most non-database applications have significantly lower scalability. In fact, many do not scale beyond two to four CPUs.
Chapter 4. Processors and cache subsystem
73
4.3.6 Processor features in BIOS BIOS levels permit various settings for performance in certain IBM System x servers. Processor Adjacent Sector Prefetch When this setting is enabled, (enabled is the default for most systems), the processor retrieves both sectors of a cache line when it requires data that is not currently in its cache. When it is disabled, the processor will only fetch the sector of the cache line that includes the data requested. For instance, only one 64-byte line from the 128-byte sector will be prefetched with this setting disabled. This setting can affect performance, depending on the application running on the server and memory bandwidth utilization. Typically, it affects certain benchmarks by a few percent, although in most real applications it will be negligible. This control is provided for benchmark users who want to fine-tune configurations and settings. Processor Hardware Prefetcher When this setting is enabled, (disabled is the default for most systems), the processors is able to prefetch extra cache lines for every memory request. Recent tests in the performance lab have shown that you will get the best performance for most commercial application types if you disable this feature. The performance gain can be as much as 20% depending on the application. For high-performance computing (HPC) applications, we recommend you turn HW Prefetch enabled and for database workloads, we recommend you leave the HW Prefetch disabled. Both prefetch settings do decrease the miss rate for the L2/L3 cache when they are enabled but they consume bandwidth on the front-side bus which can reach capacity under heavy load. By disabling both prefetch settings, multi-core setups achieve generally higher performance and scalability.
74
Tuning IBM System x Servers for Performance
In single-core processor setups, it is generally more optimal to enable the adjacent sector prefetch. Figure 4-16 shows the gains that were measured with the various settings on a tuned single core x3850 server running an online transaction processing workload.
Performance
Processor prefetch settings comparison on x366 4-way 3.66 GHz/1 MB L2, 64 GB Memory
1.0%
4.5% -0.7%
Hardware Prefetch Enabled
Both Enabled
Both Disabled Adjacent Sector Prefetch Enabled
Figure 4-16 Prefetch settings on x3850 (System x366) with single core processors
For dual-core processor configurations, it is generally more optimal to disable the prefetch settings. Figure 4-17 shows the gains that were measured with the various settings on a tuned dual core x3850 server running an online transaction processing workload.
Performance
Processor prefetch settings comparison on x3850 4-way 3.0 GHz dual core, 64 GB Memory
22% -9.0%
Both Enabled
Both Disabled
Adjacent Sector Prefetch Enabled
Figure 4-17 Prefetch settings on x3850 with dual core processors
Chapter 4. Processors and cache subsystem
75
4.4 Rules of thumb This section discusses simple rules of thumb for the CPU, L2 cache, and CPU scalability. An improvement in system performance gained by a CPU upgrade can be achieved only when all other components in the server are capable of working harder. Compared to all other server components, the Intel processor has experienced the largest performance improvement over time and in general, the CPU is much faster than every other component. A common belief is that a faster CPU will always improve system performance. In many cases, the CPU is the least likely server component to cause a bottleneck. Often, upgrading the CPU with the same number of cores simply means the system runs with lower CPU utilization, while other bottlenecked components become even more of a bottleneck. Upgrading a processor from a single core to a lower frequency dual core might even introduce a bottleneck if the software is unable to handle the processor threads. In general, the largest system performance improvements are achieved by adding memory and disk drives. If you do have a rare case of a CPU bottleneck, but have sufficient memory and LAN and disk speed, then refer to Table 4-5 on page 77 and Table 4-6 on page 78 for the approximate relative CPU performance indicator for various processor configurations. For example, if you are currently running 60.5K operations per hour with 2.0 GHz 1 MB L3 Gallatin processors and you would like to determine the possible gain from switching to 2.7 GHz 2 MB L3 Gallatin processors, you could use Table 4-6 on page 78. By looking at the table, you notice that the “Relative Indicator of Performance” for the 2.0 GHz 1 MB L3 Gallatin processor is 1.10 and the “Relative Indicator of Performance” for the 2.7 GHz 2 MB L3 Gallatin processor is 1.29. Using those values can determine the solution by a two step process. 1. Derate to a baseline score by taking 60.5 K / 1.10 = 55 K Operations/Baseline. 2. Multiply the operations/Baseline by the “Relative Indicator of Performance” by 55K * 1.29 = 71.0K operations per hour. Thus, increasing the processors frequency and cache size for this example increased the application performance from 60.5 K to 70 K. Comparing the various Xeon DP, MP processors in servers running Windows 2003 or Linux with 2.6 kernel, good rules of thumb for improvements from processor frequency and processor cache contribution to performance are shown in the following tables:
76
Tuning IBM System x Servers for Performance
Table 4-5 Xeon DP processor - approximate relative indicator of performance rules of thumb Xeon DP processor speed, cache size, FSB speed
Gain from MHz
Gain from cache
Gain from front-side bus
Relative indicator of performance
2.0 GHz 512 KB L2 400 MHz FSB - base processor
1.00
1.00
1.00
1.00
2.2 GHz 512 KB L2 400 MHz FSB
1.04
1.00
1.00
1.04
2.4 GHz 512 KB L2 400 MHz FSB
1.08
1.00
1.00
1.08
2.6 GHz 512 KB L2 400 MHz FSB
1.08
1.00
1.00
1.09
2.8 GHz 512 KB L2 400 MHz FSB
1.11
1.00
1.00
1.11
2.8 GHz 512 KB L2 533 MHz FSB
1.11
1.00
1.10
1.22
3.06 GHz 512 KB L2 533 MHz FSB
1.15
1.00
1.10
1.26
3.06 GHz 512 KB L2 1 MB L3 533 MHz FSB
1.15
1.10
1.10
1.39
3.2 GHz 512 KBL2, 1 MB L3 533 MHz FSB
1.17
1.10
1.10
1.41
3.2 GHz 512 KBL2, 2 MB L3 533 MHz FSB
1.17
1.18
1.10
1.52
EM64T architecture: 3.6 GHz 1 MB L2 800 MHz FSB - base processor
1.62
2.8 GHz 1 MB L2 800 MHz FSB
1.11
1.08
1.22
1.46
3.0 GHz 1 MB L2 800 MHz FSB
1.14
1.08
1.22
1.50
3.2 GHz 1 MB L2 800 MHz FSB
1.17
1.08
1.22
1.54
3.4 GHz 1 MB L2 800 MHz FSB
1.19
1.08
1.22
1.57
3.6 GHz 1 MB L2 800 MHz FSB
1.21
1.08
1.22
1.59
3.0 GHz 2 MB L2 800 MHz FSB
1.14
1.16
1.22
1.61
3.2 GHz 2 MB L2 800 MHz FSB
1.17
1.16
1.22
1.65
3.4 GHz 2 MB L2 800 MHz FSB
1.19
1.16
1.22
1.68
3.6 GHz 2 MB L2 800MHz FSB
1.21
1.16
1.22
1.71
Chapter 4. Processors and cache subsystem
77
Table 4-6 Xeon MP processor - approximate relative indicator of performance rules of thumb Xeon MP processor speed, cache size, code name
Gain from MHz
Gain from cache
Gain from front-side bus
Relative indicator of performance
1.6 GHz 1 MB L3 baseline processor (Foster)
1.00
1.00
1.00
1.00
1.9 GHz 1 MB L3 (Gallatin)
1.08
1.00
1.00
1.08
2.0 GHz 1 MB L3 (Gallatin)
1.10
1.00
1.00
1.10
2.5 GHz 1 MB L3 (Gallatin)
1.16
1.00
1.00
1.16
2.0 GHz 2 MB L3 (Gallatin)
1.10
1.08
1.00
1.19
2.2 GHz 2 MB L3 (Gallatin)
1.15
1.08
1.00
1.24
2.7 GHz 2 MB L3 (Gallatin)
1.19
1.08
1.00
1.29
2.8 GHz 2 MB L3 (Gallatin)
1.21
1.08
1.00
1.31
3.0 GHz 4 MB L3 (Gallatin)
1.25
1.19
1.00
1.48
2.8 GHz/4 MB L3 (Potomac)
1.21
1.15
1.10
1.53
3.0 GHz/8 MB L3 (Potomac)
1.25
1.18
1.10
1.63
3.3 GHz/8 MB L3 (Potomac)
1.31
1.18
1.10
1.70
3.16 GHz/1 MB L2 (Cranford)
1.68
3.66 GHz/1 MB L2 (Cranford)
1.77
3.0 GHz/2x2 MB Dual Core Paxville
2.52
78
Tuning IBM System x Servers for Performance
5
Chapter 5.
Virtualization hardware assists Until recently, all virtualization on x86 architecture was implemented in software. However, Intel and AMD have developed hardware virtualization technology that is designed to: Allow guest operating systems, VMMs, and applications to run at their standard privilege levels. Eliminate the need for binary translation and paravirtualization. Provide more reliability and security. The first phase of this iterative development endeavor was implemented in the processors. Intel has named their hardware virtualization technology VT-x. AMD has named their hardware virtualization technology Pacifica. In the future, additional phases of hardware virtualization technology will be developed for I/O and memory. This chapter includes the following sections: 5.1, “Introduction to virtualization technology” on page 80 5.2, “Virtualization hardware assists” on page 82 5.3, “Support for virtualization hardware assists” on page 84
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
79
5.1 Introduction to virtualization technology Note: for more information regarding virtualization, see the IBM Redbook Virtualization on the IBM System x3950 Server, SG24-7190. This chapter provides an introduction to virtualization hardware assists. It touches on some virtualization concepts, but it is not intended to provide a lengthy review of virtualization concepts. Conventionally, a server is loaded with a single operating system that controls access to the server’s hardware resources such as the processors, memory, and I/O related technology. Virtualization enables multiple operating systems, called guest operating systems, to run on a single server and to share access to the server’s hardware resources. In order to share resources between multiple guest operating systems, a software virtualization layer is required to manage the utilization of the resources by each guest operating system. Figure 5-1 illustrates a guest operating system that is running on a server with a software virtualization layer between the key hardware resources and the guest operating systems.
(Standard) guest OS Standard x86 hardware calls
Full x86 hardware virtualized interface Hypervisor
x86 hardware
Figure 5-1 Full virtualization architecture
5.1.1 Privilege levels There are challenges with the virtualization model in Figure 5-1 on an x86 architecture. On an x86 architecture without virtualization, the operating system is designed to run at privilege level 0, which is the most powerful privilege level.
80
Tuning IBM System x Servers for Performance
Privilege level 0 gives the operating system access to the hardware resources of a server, so that it can execute instructions to obtain the state of the hardware resources and control those resources. Applications tend to run at privilege level 3 and do not have direct access to the hardware. When the software virtualization layer is introduced, the guest operating system is bumped to privilege level 1, and the virtualization layer runs at privilege level 0. This shift from a more powerful privilege to a less powerful privilege level is called ring deprivileging. Ring deprivileging can introduce faults, because many Intel architecture instructions which control CPU functionality were designed to be executed from privilege level 0, not privilege level 1. Therefore, the software virtualization layer must trap certain instructions and hardware accesses and then emulate those instructions back to the guest operating system. These additional steps introduce more complexity and, therefore, the chance of more faults. If a guest operating system needs to access memory, the virtualization layer must intercept, interpret, execute, and then return the result back to the guest operating system. In addition, the virtualization layer must handle all interrupts to and from the guest operating system. A guest operating system can decide whether to block or to permit interrupts, depending on the operation in progress. The virtualization layer must be able to track what the guest operating system decides to do with an interrupt, which adds overhead to system resources. The overhead produces a performance penalty.
5.1.2 Binary translation and paravirtualization Binary translation is a software method used to address the challenges of ring deprivileging. Both VMware and Microsoft virtualization products use binary translation. The hypervisor traps certain privileged guest operating system instructions, and then translates them into instructions that the virtualized environment can execute. When a privileged instruction is received by the hypervisor, it takes control of the required hardware resource, resolves any conflicts, and returns control to the guest operating system. If the hypervisor functions correctly, the guest operating system does not know that the hypervisor is emulating the hardware resource. However, binary translation requires specific builds of operating systems. A new release of the operating system requires testing and perhaps changes to the hypervisor code.
Chapter 5. Virtualization hardware assists
81
Paravirtualization is another software solution to the challenges of ring deprivileging. It is used by Xen, an open source virtualization project. In this case, the operating system source code is altered so that it can call the hypervisor directly to perform low-level functions. Figure 5-2 illustrates the paravirtualization architecture.
(Paravirtualized) guest OS Specific virtual-aware hypervisor calls Standard x86 hardware calls
Full x86 hardware virtualized interface Hypervisor
x86 hardware
Figure 5-2 Paravirtualization architecture
Paravirtualization reduces the complexity of the virtualization software layer, and it can improve performance. In this case, the modified guest OS is virtual-aware and shares the virtualization load with the hypervisor. This reduces the complexity of the hypervisor, which can be implemented more efficiently. However, off-the-shelf operating systems cannot be used with paravirtualization. Tip: See 12.11, “Xen virtualization” on page 420, for more information about Xen virtualization.
5.2 Virtualization hardware assists Virtualization hardware assists have been developed and will continue to be developed to overcome the challenges and complexities of software virtualized solutions presented in the previous sections of this chapter. Intel’s virtualization hardware assist is called Intel Virtualization Technology, Intel VT. Intel VT-x provides hardware support for IA32 and 64-bit Xeon processor virtualization.
82
Tuning IBM System x Servers for Performance
Note: VT-i provides hardware support for Itanium processor virtualization; however, the discussion of VT-i is outside the scope of this redbook. VT-x introduces two new CPU operations: VMX root operation—VMM functions VMX non-root operation—guest operating system functions Both new operations support all four privilege levels. This support allows a guest operating system to run at privilege level 0 where an operating system is designed to run on an x86 platform. VT-x also introduces two transitions: VM entry is defined as a transition from VMX root operation to VMX non-root operation or VMM to guest operating system. VM exit is defined as a transition from VMX non-root operation to VMX root operation or from guest operating system to VMM. VT-x also introduces a new data structure called the virtual-machine control structure (VMCS). The VMCS tracks VM entries and VM exits, as well as, processor state of guest operating system and VMM in VMX non-root operations. AMD’s virtualization hardware, Pacifica, introduces a new processor mode called Guest Mode. Guest Mode is similar to the VMX non-root mode in Intel VT. A new data structure called the virtual machine control block (VMCB) tracks the CPU state for a guest operating system. If a VMM wants to transfer processor control to a guest operating system, it executes a VMRUN command. A VMRUN entry is analogous to the Intel VT VM entry transition. A VMRUN exit is analogous to the Intel VT VM exit transition. Studies and measurements indicate that the first pass of virtualization hardware assists which were implemented in the processor as described in this section did not significantly improve performance. It appears as though the software implementations of binary-translation and paravirtualization are still the best performing methods to implement privileging. This is not a complete surprise, because processor utilization does not tend to be the largest area of concern with respect to virtualization performance. In the future, look for hardware based page tables implemented in processors that will replace shadow page tables currently implemented in software, which should decrease hypervisor overhead. In addition, look for I/O hardware assists, which tend to be one of the largest areas of concern with respect to virtualization
Chapter 5. Virtualization hardware assists
83
performance. These two areas will be addressed in the next passes of virtualization hardware assists.
5.3 Support for virtualization hardware assists For using virtualization hardware assists, you must have: BIOS enablement Hardware technology Hypervisor support Important: System BIOS enablement, hardware technology (for example, technology within the processors for first generation hardware assists), and Hypervisor support are all required to implement virtualization hardware assists. Processor virtualization hardware assists are included in the following Intel processors:
Dempsey Woodcrest Paxville Tulsa
Processor virtualization hardware assists are included in Rev F processors from AMD. Keep in mind, the end result of these virtualization hardware assists is to reduce the complexity of the hypervisor, which can reduce overhead and improve performance significantly.
84
Tuning IBM System x Servers for Performance
6
Chapter 6.
PCI bus subsystem The Peripheral Component Interconnect (PCI) bus is the predominant bus technology that is used in most Intel architecture servers. The PCI bus is designed to allow peripheral devices, such as LAN adapters and disk array controllers, independent access to main memory. PCI adapters that have the ability to gain direct access to system memory are called bus master devices. Bus master devices are also called direct memory access (DMA) devices. This chapter discusses the following topics:
6.1, “PCI and PCI-X” on page 86 6.2, “PCI-X” on page 86 6.3, “PCI Express” on page 90 6.4, “Bridges and buses” on page 94
To simply this chapter, we have combined our discussion of PCI and PCI-X into one section and have outlined any differences between the two standards.
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
85
6.1 PCI and PCI-X The PCI bus is designed as a synchronous bus, meaning that every event must occur at a particular clock tick or edge. The standard PCI bus uses a 33 MHz or 66 MHz clock that operates at either 32-bit or 64-bit. With the introduction of PCI-X, the speeds have been increased to include 66 MHz, 133 MHz, 133 MHz DDR, and 133 MHz QDR. This increase has raised the maximum transfer rate in burst mode from 276 MBps to 4.2 GBps. PCI uses a multi-drop parallel bus that is a multiplexed address and data bus, meaning that the address and data lines are physically the same wires. Thus, fewer signal wires are required, resulting in a simpler, smaller connector. The downside to this design is that PCI transactions must include a turnaround phase to allow the address lines to be switched from address mode to data mode. The PCI bus also has a data-pacing mechanism that enables fast devices to communicate with slower devices that are unable to respond to a data transfer request on each clock edge. The generic name for any PCI device is the agent. A basic data transfer operation on the PCI bus is called a PCI transaction, which usually involves request, arbitration, grant, address, turnaround, and data transfer phases. PCI agents that initiate a bus transfer are called initiators, while the responding agents are called targets. All PCI operations are referenced from memory. For example, a PCI read operation is a PCI agent reading from system memory. A PCI write operation is a PCI agent writing to system memory. PCI transactions do not use any CPU cycles to perform the transfer. The language of PCI defines the initiator as the PCI bus master adapter that initiates the data transfer (for example, a LAN adapter or SCSI adapter) and the target as the PCI device that is being accessed. The target is usually the PCI bridge device or memory controller.
6.2 PCI-X PCI-X 2.0 is the latest version of PCI and is built upon the same architecture, protocols, signals, and connectors as traditional PCI. This architecture has resulted in maintaining hardware and software compatibility with the previous generations of PCI. This design means that devices and adapters that are compliant with PCI-X 1.0 are fully supported in PCI-X 2.0. When supporting previous PCI devices, it is important to note that the clock must scale to a frequency that is acceptable to the lowest speed device on the bus. This results in all devices on that bus being restricted to operating at that slower speed.
86
Tuning IBM System x Servers for Performance
PCI-X was developed to satisfy the increased requirements of today’s I/O adapters, such as Gigabit Ethernet, Fibre Channel, and Ultra320 SCSI. PCI-X is fully compatible with standard PCI devices. It is an enhancement to the conventional PCI specification V2.2 and enables a data throughput of over 4 GBps at 533 MHz/64-bits in burst mode. Adapters with high I/O traffic, such as Fibre Channel and storage adapters, benefit significantly from PCI-X. These adapters provide a huge amount of data to the PCI bus and, therefore, need PCI-X to move the data to main memory. Tip: Simply migrating to a newer PCI bus might not alleviate the bottleneck in a system. Although the peak throughput has increased from PCI to PCI-X, this is not the only reason why PCI-X shows increased performance over PCI adapters. The following are changes made in PCI-X that provide higher efficiency and, therefore, a performance benefit when compared to standard PCI: Attribute phase The attribute phase takes one clock cycle and provides further information about the transaction. PCI-X sends new information with each transaction performed within the attribute phase which enables more efficient buffer management. The attribute phase can be split into several parts: – Sequence information: Each transaction in a sequence identifies the total number of bytes remaining to be read or written. If a transaction is disconnected, the new transaction that continues the sequence includes an updated byte count. Furthermore, each transition includes the identity of the initiator (bus number, device number, and function number) and additional information about transaction ordering and cache requirements. – Relaxed order structure: Relaxed ordering is a technique that allows PCI-PCI bridges to rearrange the transactions on the bus. Therefore, more important data is transmitted before less important data, and the efficiency of the system improves. – Transaction byte count: With this information, the PCI-PCI bridge gets information about how long a transaction will take. For every transaction, the byte count holds a count of how much data is remaining and, therefore, enables the PCI-PCI bridge to use its internal cache more efficiently. Split transactions Delayed transactions in conventional PCI are replaced by split transactions in PCI-X. All transactions except memory-write transactions are allowed to be executed as split transactions. If a target on the PCI bus cannot complete a transaction within the target initial latency limit, the target must complete the
Chapter 6. PCI bus subsystem
87
transaction as a split transaction. Thus, the target sends a split response message to the initiator telling it that the data will be delivered later on. This the frees the bus for other communications. When the data is available for transmission, the target requests access to the bus and completes the transaction with split completion transaction. For example, a SCSI controller that is waiting for data from a disk and is blocking the PCI bus for other devices is forced to complete the transaction as a split transaction. If the target meets the target initial latency limits, it optionally completes the transaction immediately (for example, the requested data was immediately available because it was found in the buffer of the SCSI adapter and it is immediately sent back to the initiator). The split transaction design replaces the similar, but less efficient delayed transactions used by older PCI specifications. The transactions are tagged and queued, and the specifications also allow for a relaxed ordering scheme that makes out-of-order execution possible. Allowable disconnect boundary When a burst transaction is initiated to prevent a single process from monopolizing the bus with a single large transfer (bursts can be up to 4096 bytes), PCI-X gives initiators and targets the chance to place interruptions. The interruptions are not placed randomly (which might compromise the efficiency of the buffers and cache operations) but are fixed on 128-byte boundaries — a figure big enough to facilitate complete cache line transmissions. The reliability of the PCI-X bus is improved by differentiating between peripheral and system errors. PCI-X has no possibility of recovering from system errors, but the device generating the error can be held in reset status, keeping the rest of the system up and running. The benefit of adopting the PCI-X standard is the increase in supported throughputs, evident with the 533 MHz implementation. When running at higher frequencies (133 MHz and higher), only one device can be on a PCI-X bus, making PCI-X a high-bandwidth point-to-point I/O channel. At lower speeds (less than 133 MHz), multiple devices can be connected on a single bus.
88
Tuning IBM System x Servers for Performance
Note that the 66 MHz implementation of PCI-X doubles the number of slots supported on a current PCI 2.2 66 MHz bus. Table 6-1 shows the possible combinations of PCI modes and speeds. Table 6-1 PCI and PCI-X modes Mode
PCI Voltage (V)
64-bit
32-bit
16-bit
Max slots
MBps
Max slots
MBps
MBps
PCI 33
5 or 3.3
4
266
4
133
Not applicable
PCI 66
3.3
2
533
2
266
Not applicable
PCI-X 66
3.3
4
533
4
266
Not applicable
PCI-X 133a
3.3
2
800
2
400
Not applicable
PCI-X 133
3.3
1
1066
1
533
Not applicable
PCI-X 266
3.3 or 1.5
1
2133
1
1066
533 MBps
PCI-X 533
3.3 or 1.5
1
4266
1
2133
1066 MBps
a. Operating at 100 MHz
PCI-X devices use 3.3V I/O signalling when operating in PCI-X mode. They also support the 5V I/O signalling levels when operating in 33 MHz conventional mode, which results in cards either designed specifically for 3.3V PCI-X or universally keyed.
3.3V slot
5v slot
64-bit slot and connector
Figure 6-1 Adapter keying
PCI-X cards are designed to run at either 66 MHz or 133 MHz. PCI-X cards are not designed usually to run at 100 MHz. However, the number of the loads on the bus can force a 133 MHz adapter to operate at 100 MHz.
Chapter 6. PCI bus subsystem
89
6.2.1 Performance It is rare for the PCI bus to be able to sustain the maximum theoretical throughput rates that are shown in Table 6-1 on page 89. In most servers, the sustainable PCI throughput is only about 75% of the maximum theoretical rate. Of course, if the width of the PCI bus doubles or if the peak speed of the bus doubles, then the maximum throughput will increase accordingly. The PCI adapter, the adapter device driver, and the system PCI chipset all limit the maximum sustainable throughput. The device driver and adapter firmware play a role in how the adapter is programmed to transfer data over the PCI bus. In most cases, 75% bus efficiency is typical. Every PCI transaction requires request, arbitration, grant, address, turnaround, and data transfer cycles. The non-data-transfer cycles are request, arbitration, address and turnaround. As described in 6.1, “PCI and PCI-X” on page 86, a turnaround cycle is required because the PCI bus shares the same signal lines for both data and address. In most cases, these cycles are overhead. During these cycles, the bus does not transfer data. Obviously, overhead cycles affect the sustainable data transfer rate. An adapter bursting small amounts of data for each transaction will have a higher percentage of overhead than an adapter bursting large amounts of data during each transaction and will, therefore, have a lower data throughput.
6.3 PCI Express PCI Express is the latest development in PCI to support adapters and devices. The technology is aimed at multiple market segments, meaning that it can be used to provide for connectivity for chip-to-chips, board-to-boards, and adapters. PCI Express uses a serial interface and allows for point-to-point interconnections between devices using directly wired interfaces between these connection points. This design differs from previous PCI bus architectures which used a shared, parallel bus architecture. A single PCI Express serial link is a dual-simplex connection that uses two pairs of wires — one for pair for transmit and one pair for receive — and that transmits only one bit per cycle. Although this design sounds limiting, it can transmit at the extremely high speed of 2.5 Gbps, which equates to a burst mode of 320 MBps on a single connection. This two pair of wires is called a lane. A PCI Express link is comprised of one or more lanes. In such configurations, the connection is labeled as x1, x2, x4, x12, x16, or x32, where the number is
90
Tuning IBM System x Servers for Performance
effectively the number of lanes. So, where PCI Express x1 would require four wires to connect, an x16 implementation would require 16 times that amount or 64 wires. This implementation results in physically different sized slots. Tip: When you refer to lane nomenclature, you use the word by, as in by 8 for x8. Figure 6-2 shows the slots for a 32-bit PCI 2.0, PCI Express x1 and a PCI Express x16. From this figure, it is clear that the PCI Express x16 adapter will not fit physically in the PCI x1 slot. Slots: PCI 2.0 32-bit
PCI 2.0 32-bit
PCI Express x1
PCI Express x16
Figure 6-2 PCI 2.0 and PCI Express edge connectors
You can install PCI Express slots in larger slots but not in smaller ones. For example, you can install a PCI Express x8 adapter into an x16 slot (although it will still operate at the x8 speed), but you cannot insert an x8 adapter into an x4 slot. Table 6-2 on page 92 shows this compatibility.
Chapter 6. PCI bus subsystem
91
Table 6-2 PCI Express slot compatibility x1 slot
x4 slot
x8 slot
x16 slot
x1 card
Supported
Supported
Supported
Supported
x4 card
No
Supported
Supported
Supported
x8 card
No
No
Supported
Supported
x16 card
No
No
No
Supported
Typically, the size of a slot matches the number of lanes it has. For example, a x4 slot typically is a x4 link (that is, it has 4 lanes). However this is not always the case. The PCI Express specification allows for the situation where the physical connector is larger than the number of lanes of data connectivity. The only requirement on manufacturers is that the connector must still provide the full complement of power and ground connections as required for the connector size. For example, in the System x3650, there are two pairs of slots: Two slots labelled “PCI Express x8 (x8 lanes)” Two slots labelled “PCI Express x8 (x4 lanes)” The first pair are PCI Express with x8 physical connectors (in other words, they will physically accept x8 cards, as well as x4, x2 and x1 cards), and they have the bandwidth of a x8 link (8x 2.5 Gbps or 20 Gbps). The second pair are also PCI Express with x8 physical connectors, but only have the bandwidth of a x4 link (4x 2.5 Gbps or 10 Gbps). If you have a need for x8 bandwidth (such as for an Infiniband or Myrinet adapter), then ensure you select one of the correct slots (the ones with x8 lanes). It is important to understand this naming convention as it will have a direct impact on performance if you select a slot that is slower than the maximum supported by the adapter. Tip: The physical size of a PCI Express slot is not the sole indicator of the possible bandwidth of the slot. You must determine from slot descriptions on the system board or the service label of the server what the bandwidth capacity is of each slot. While the underlying hardware technology is different between PCI-X and PCI Express, they remain compatible at the software layer. PCI Express supports existing operating systems, drivers, and BIOS without changes. Because they are compatible at the level of the device driver model and software stacks, PCI Express devices look just like PCI devices to software.
92
Tuning IBM System x Servers for Performance
A benefit of PCI Express is that it is not limited for use as a connector for adapters. Due to its high speed and scalable bus widths, you can also use it as a high-speed interface to connect many different devices. You can use PCI Express to connect multiple onboard devices and to provide a fabric that is capable of supporting USB 2, InfiniBand, Gigabit Ethernet, and others.
6.3.1 PCI Express performance PCI Express currently runs at 2.5 Gbps or 200 MBps per lane in each direction, providing a total bandwidth of 80 Gbps in a 32-lane configuration and up to 160 Gbps in a full duplex x32 configuration. Future frequency increases will scale up total bandwidth to the limits of copper (which is 12.5 Gbps per wire) and significantly beyond that through other media without impacting any layers above the physical layer in the protocol stack. Table 6-3 shows the throughput of PCI Express at different lane widths. Table 6-3 PCI Express maximum transfer rate Lane width
Clock speed
Throughput (duplex, bits)
Throughput (duplex, bytes)
Initial expected uses
x1
2.5 GHz
5 Gbps
400 MBps
Slots, Gigabit Ethernet
x2
5 GHz
10 Gbps
800 MBps
None
x4
10 GHz
20 Gbps
1.6 GBps
Slots, 10 Gigabit Ethernet, SCSI, SAS
x8
20 GHz
40 Gbps
3.2 GBps
Slots, Infiniband adapters, Myrinet adapters
x16
40 GHz
80 Gbps
6.4 GBps
Graphics adapters
PCI Express uses an embedded clocking technique that uses 8b/10b encoding. The clock information is encoded directly into the data stream, rather than having the clock as a separate signal. The 8b/10b encoding essentially requires 10 bits per character or about 20% channel overhead. This encoding explains differences in the published specification speeds of 250 MBps (with the embedded clock overhead) and 200 MBps (data only, without the overhead). For ease of comparison, Table 6-3 shows throughput in both bps and Bps. When compared to the current version of a PCI-X 2.0 adapter running at 133 MHz QDR (quad data rate, effectively 533 MHz), the potential sustained throughput of PCI Express x16 is over double the throughput as shown in Figure 6-3 on page 94.
Chapter 6. PCI bus subsystem
93
PCI Express 80 Gbps
x16 x8
40 Gbps 2 Gbps0
x4
10 Gbps
x2 x1
5 Gbps
PCI-X 2.0 (64-bit 266/533 MHz) QDR
32 Gbps
DDR
16 Gbps
PCI-X 1.0 (64-bit 133 MHz) 8 Gbps PCI 2.3 (64-bit 66 MHz) 4 Gbps PCI 1.0 (32-bit 33 MHz) 1 Gbps
Figure 6-3 PCI Express and PCI-X comparison (in Gbps)
6.4 Bridges and buses When PCI first appeared on the market, systems were limited to two or three PCI slots. This limitation was due to the signal limitations of the PCI bus. To overcome this limitation, the concept of the PCI-to-PCI (PtP) bridge was developed. Early implementations of the PtP bridge involved a primary bus and a secondary bus. Access to devices on the secondary bus was typically slower as the I/O requests negotiated the bridge. Modern PCI designs, such as that used in the System x3950, employ multiple PCI bridges to provide PCI buses at a variety of speeds, as shown in Table 6-4 on page 95.
94
Tuning IBM System x Servers for Performance
CPU 1
CPU 2
667 MHz 5.33 GBps
DDR2
SMI2
DDR2
SMI2
DDR2
SMI2
DDR2
SMI2
Each: 667 MHz 5.33 GBps
667 MHz 5.33 GBps
266 266 MHz
IBM XA-64e core chipset
Scalability ports Each: 6.4 GBps
6 GBps
6 GBps
PCI-X bridge 33 66
CPU 4
Memory controller ("Hurricane")
6 GBps
Calgary
CPU 3
PCI-X bridge 266 MHz
Video RSA SL 1 USB 2.0
ServeRAID
South bridge
Adaptec SAS
EIDE
K/M
2
3
4
5
6
HDD backplane
Gigabit Ethernet
Six PCI-X 2.0 slots: 64-bit 266 MHz
Serial
Figure 6-4 IBM System x3950 block diagram
Because PCI Express is point-to-point, as opposed to multiplexed parallel, the requirement to interface with multiple edge connectors through a bridge does not exist. In essence, the PCI Express slot interfaces directly with the memory controller through a series of channels. This type of interface means that bandwidth to the edge connectors does not need to be managed in the same way. With PCI-X, the aggregate speed of the edge connectors cannot exceed the allocated bandwidth between the memory controller and the PCI bridge. This places a limitation on the number and combinations of speeds of PCI slots that can be supported on a single bridge. Removing the requirement to connect PCI cards through a bridge also reduces latency because the data has one less hop to travel.
Chapter 6. PCI bus subsystem
95
Note: It is important to remember that the primary function is to transfer data from the adapter into memory (through DMA) as quickly as possible so that it can be processed by the CPU. PCI Express achieves transfers data more quickly by reducing the number of hops to memory and by increasing throughput. Figure 6-5 illustrates how the PCI Express slots connect directly to the memory controller while the PCI-X edge connectors connect to the memory controller through a PCI bridge. The x3650 implements these slots on replaceable riser cards.
CPU
CPU 4-way interleaved Fully buffered DIMMs
Blackford Memory controler
PCI Express
ServeRAID SAS South bridge
HDD backplane External port
PCI Express x8 PCI Express x8 USB 2.0 PCI Express x4 Video
Replaceable riser cards
PCI Express x4 Serial Gigabit Ethernet
PCI Bridge
133 MHz PCI-X 133 MHz PCI-X
Figure 6-5 System x3650 block diagram with PCI Express or PCI-X riser cards
96
Tuning IBM System x Servers for Performance
7
Chapter 7.
Chipset architecture The chipset architecture implements the control and data flow between the processor, memory, PCI devices, and system buses. Chipsets are varied in functionality and performance bottlenecks. Other functions such as video, keyboard, interrupt, diskette, and clock are provided by support chips. This chapter discusses the following topics: 7.1, “Overview of chipsets” on page 98 7.2, “System architecture design and performance” on page 100 7.3, “Memory controller-based chipset” on page 110 – – – – –
7.3.1, “ServerWorks Grand Champion 4.0 HE and LE” on page 111 7.3.2, “Intel E7520 and E7525” on page 112 7.3.3, “Intel 5000 chipset family” on page 114 7.3.4, “XA-64e third generation chipset” on page 116 7.3.5, “Intel E8500 Chipset” on page 121
7.4, “PCI bridge-based chipsets” on page 123
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
97
7.1 Overview of chipsets While processor performance has been increasing rapidly, improvements to memory have not been as dramatic. Increases in the working set sizes for software have caused larger memory footprints which in turn have necessitated larger caches and main memory. A by-product of the increased cache size is a higher latency to access main memory. The chipset controls the flow of data between the processors, external caches, and memory which makes the chipset an integral part in controlling the system wide latency. System and chip designers generally use a key metric known as Cycles Per Instruction (CPI) to measure the number of processor clocks that a system uses to execute an instruction. While the number of instructions to execute an operation is held constant, a decrease in the number of cycles combined with a higher clock rate provides a measurable increase in performance. Many workloads, particularly the random nature of workloads that are prevalent in servers, have frequent cache misses. As a result, a greater component of the CPI for server class workloads are dependent on the chipset and memory subsystem then the core processor. Thus, the chipset is a major contributor to the overall performance to a system. Table 7-1 lists the chipsets that IBM System x and BladeCenter servers use. We discuss the chipsets that are listed in bold font in detail in this chapter. Table 7-1 Chipsets that System x and BladeCenter servers use Server
Chipset
xSeries servers
98
x100
Intel E7230
x206m
Intel E7230
x225
Intel E7505
x226
Intel E7525 (page 112)
x235
ServerWorks Grand Champion 4.0 LE (page 111)
x236
Intel E7520 (page 112)
x260
IBM XA-64e 3rd generation chipset (page 116)
x305
ServerWorks Grand Champion SL
x306m
Intel E7210
e326m
ServerWorks HT2000 system I/O controller
Tuning IBM System x Servers for Performance
Server
Chipset
x335 DC
ServerWorks Grand Champion 4.0 LE (page 111)
x336
Intel E7520 (page 112)
x343
Intel E7501
x343DC
ServerWorks III HE-SL
x346
Intel E7520 (page 112)
x360
IBM XA-32
x366
IBM XA-64e 3rd generation chipset (page 116)
x445
IBM XA-32
x460
IBM XA-64e 3rd generation chipset (page 116)
System x servers x3400
Intel 5000P chipset (page 114)
x3455
Broadcom HT2100 PCI-E bridge chip
x3500
Intel 5000P chipset (page 114)
x3550
Intel 5000X chipset (page 114)
x3650
Intel 5000P chipset (page 114)
x3650T
Intel E7520 (page 112)
x3755
ServerWorks HT2100 PCI-E bridge chip
x3800
IBM XA-64e 3rd generation chipset (page 116)
x3850
IBM XA-64e 3rd generation chipset (page 116)
x3950
IBM XA-64e 3rd generation chipset (page 116)
BladeCenter servers HS20
Intel E7520 (page 112)
HS40
ServerWorks Grand Champion 4.0 LE (page 111)
JS20
AMD 8131 HyperTransport Tunnel and PCI-X Bridge
JS21
BCM5780 PCI-E, HyperTransport Tunnel
LS20
AMD 8111 HyperTransport Bridge Chip
LS21 / LS41
ServerWorks HT-2000 HT
Chapter 7. Chipset architecture
99
Tip: The System x Reference (xREF) is a set of one-page specification sheets for each of the System x server models and includes details of the chipsets used. xREF is available from: http://www.ibm.com/servers/eserver/education/cust/xseries/xref.html
7.2 System architecture design and performance Today's server workloads tend to grow larger and larger and require more and more CPU and memory resources. Depending on the application, you have two options to answer the increasing demand: Scaling up (one larger server) Scaling out (many smaller servers) Several applications allow you to scale out. A typical example is Web site hosting. Large Web sites are not hosted on a single large SMP server but rather are hosted by a number of one-socket or two-socket servers using a distributed workload model. In general, this method is efficient because much of the data is read-only and not shared across many concurrent sessions. This fact enables scale-out computing to experience improved performance as nodes are added because each server can provide read-only data to unique concurrent users, independent of any other machine state. There are also a number of applications, such as virtualization and database environments, that scale up. Scale-up refers to the idea of increasing processing capacity by adding additional processors, memory, and I/O bandwidth to a single server, making it more powerful. A typical scale-up server for such an application is a multi-processor system such as the System x3950. However, hardware scalability is only one aspect of building scalable multiprocessor solutions. It is paramount that the operating system, driver, and application scale just as well as the hardware. In this section, we first explore the main concepts of multi-processing and then examine software scalability.
7.2.1 Hardware scalability When considering CPUs, cache, and memory in an SMP configuration, memory is the most frequently used and also has the greatest latency of the shared hardware resources. Hardware scalability is usually defined by how efficiently CPUs share memory, because the fast CPUs must frequently access the slower memory subsystem.
100
Tuning IBM System x Servers for Performance
High-speed caches are used to accelerate access to memory objects that the CPU uses most frequently, but performance gains that are obtained by high-speed caches introduce problems that can often limit multi-processor hardware scalability. There are two architectures available in System x servers with multiple processors: SMP and NUMA.
7.2.2 SMP Most of the Intel-compatible systems are designed using an SMP architecture. Designing a system to use multiple concurrently-executing CPUs is a complex task. The most popular method is to design the system so that all CPUs have symmetric access to all hardware resources such as memory, I/O bus, and interrupts, thus the name symmetric multiprocessing. SMP is most popular because it simplifies the development of the operating system and applications. Because each CPU sees the same hardware resources, no special software techniques are needed to access any resource. Therefore, SMP hardware scalability is directly related to how efficiently the many CPUs use the shared hardware resources. One of the disadvantages of SMP is the limited scalability of this architecture. As processors are added to the system, the shared resources are frequently accessed by an increasingly greater number of processors. More processors using the same resources creates queuing delays similar to many people trying to pay in a market with just one cashier. While the single cashier is symmetric, which means that everyone has to pay in the same location making it easy to locate, the disadvantage is that everyone must wait in the same queue.
7.2.3 NUMA The non-uniform memory access (NUMA) architecture is a way of building very large multi-processor systems without jeopardizing hardware scalability. The name NUMA is not completely correct because not only memory can be accessed in a non-uniform manner but also I/O resources. NUMA effectively means that every processor or group of processors has a certain amount of memory local to it. Multiple processors or multiple groups of processors are then connected together using special bus systems (for example, the HyperTransport links in the AMD-based System x3755 or the scalability ports of the Xeon-based System x3950) to provide processor data coherency. The essence of the NUMA architecture is the existence of multiple memory subsystems, as opposed to a single one on a SMP system.
Chapter 7. Chipset architecture
101
The so-called local or near memory has the same characteristics as the memory subsystem in an SMP system. However, by limiting the number of processors that access that memory directly, performance is improved because of the much shorter queue of requests. Because each group of processors has its local memory, memory on another group of processors is considered remote to the local processor. This remote memory can be accessed but at a longer latency than local memory. All requests between local and remote memory flow over the inter-processor connection (HyperTransport or scalability ports). Consider a two node System x3950 configuration with a total of eight processors and 128 GB of memory. Each x3950 has four CPUs and 64 GB of RAM. as shown in Figure 7-1. These two systems are connected together using their 6.4 GBps scalability ports. System x3950
CPU
System x3950 CPU
CPU
CPU
CPU
CPU
CPU
CPU
Scalability ports (each 6.4 GBps)
DIMMs DIMMs
DIMMs
Memory controller
DIMMs
DIMMs
Memory controller
DIMMs
DIMMs
DIMMs
PCI-X
PCI-X
PCI-X
PCI-X
PCI-X slots
Devices & slots
Devices & slots
PCI-X slots
Figure 7-1 A two-node x3950 configuration
An application that is running on CPUs in on server node can access memory that is located physically in the other node (a remote access). This access incurs longer latency because the travel time to access remote memory on another expansion module is greater. Many people think access latency is a problem with NUMA. However, this focus on latency misses the actual issue that NUMA is attempting to solve.
102
Tuning IBM System x Servers for Performance
Another way to think about it is to imagine the following scenario. You are paying for your groceries in your favorite grocery store. Directly in front of you is a cashier with 20 customers standing in line, but 50 feet to your left is another cashier with only two customers standing in line. Which cashier would you choose? The cashier closest to your position has the lowest latency because you do not have far to travel. However, the cashier 50 feet away has much greater latency because you have to walk 50 feet. Generally, most people would walk the 50 feet and suffer the latency to arrive at a cashier with only two customers instead of 20. We think this way because our experience tells us that the time waiting to check out with 20 people ahead is far longer than the time needed to walk to the “remote” cashier and wait for only two people ahead. This analogy communicates the performance effects of queuing time versus latency. In a computer server, with many concurrent outstanding memory requests, we would gladly incur come additional latency (walking) to spread memory transactions (paying for our groceries) across multiple memory controllers (cashiers) because this improves performance greatly by reducing the queuing time. We do not want to walk 50 feet to a cashier that has 20 customers paying when one is directly in front of us with only two customers. So, to reduce unnecessary remote access, NUMA systems such as the System x3950 maintain a table of data in the firmware call the Static Resource Allocation Table (SRAT). The data in this table is accessible by operating systems such as Windows Server 2003 (Windows 2000 Server does not support it) and current Linux kernels. These modern operating systems attempt to allocate resources that are local to the processors that are used by each process. So, when a process and its threads start on node 0, all execution and memory access are local to node 0. As more processes are added to the system, the operating system balances them across the nodes. In this case, most memory accesses are evenly distributed across the multiple memory controllers, thus reducing remote access, greatly reducing queuing delays, and improving performance. The AMD Opteron implementation is called Sufficiently Uniform Memory Organization (SUMO) and is a NUMA architecture. In the case of the Opteron, each processor has its own local memory with low latency. Every CPU can also access the memory of any other CPU in the system but with some latency.
NUMA Optimization for Windows Server 2003 Most editions of Windows Server 2003 are optimized for NUMA as listed in Table 7-2 on page 104. Windows Server 2003 obtains the NUMA information from the SRAT table in the system BIOS while booting. That is, NUMA
Chapter 7. Chipset architecture
103
architecture servers must have the SRAT table to use this function. Windows Server 2003 cannot recognize system topology without the SRAT table. Table 7-2 Versions of Windows Server optimized for NUMA x86 (32-bit)
x64 (64-bit)
IA64 (64-bit)
Windows 2003 Web Edition
No
Not applicable
Not applicable
Windows Server 2003 Standard Edition
No
No
Not applicable
Windows Server 2003 Enterprise Edition
Yes
Yes
Yes
Windows Server 2003 Datacenter Edition
Yes
Yes
Yes
NUMA Optimization in Linux The 2.6 kernel features NUMA awareness in the scheduler (the part of the operating system that assigns system resources to processes) so that the vast majority of processes execute in local memory. This information is passed to the operating system through the ACPI interface and the SRAT table similar to the Windows Operating System.
Static Resource Affinity Table The Static Resource Affinity Table (SRAT) includes topology information for all the processors and memory in a system. The topology information includes the number of nodes in the system and which memory is local to each processor. By using this function, the NUMA topology recognized by the operating system. The SRAT table also includes hot-add memory information. Hot-add memory is the memory that can be hot-added while the system is running, without requiring a reboot. The Advanced Configuration and Power Interface (ACPI) 2.0 specification introduces the concept of proximity domains in a system. Resources, including processors, memory, and PCI adapters in a system, are tightly coupled, and the operating system can use this information to determine the best resource allocation and the scheduling of threads throughout the system. The SRAT table is based on this ACPI specification. You can find more about the SRAT table at: http://www.microsoft.com/whdc/system/CEC/SRAT.mspx The SRAT table is automatically configured in systems such as the x3950 in firmware. For other systems, you should enable the SRAT information in the system BIOS (if this is configurable) and run a NUMA-aware operating system. Keep in mind that many applications require at least two to four processors to reach maximum performance. In this case, even with NUMA-aware operating
104
Tuning IBM System x Servers for Performance
systems, there can be a high percentage of remote memory accesses in an Opteron system because each processor is the only processor on a node. The frequency of NUMA access depends upon the application type and how users apply that application and cannot be estimated without extensive analysis. Tip: In the IBM eServer™ 326 system BIOS, enable the ACPI SRAT, disable Node Interleave, and set the DRAM Interleave to Auto to achieve the best performance in conjunction with a NUMA-aware operating system.
7.2.4 The MESI protocol A complicating factor for the design of any SMP system is the need to keep all CPU and cache data coherent. Because each CPU has access to the same data that is stored in memory, two or more CPUs should not modify the same data at the same time. This concurrent modification of the same data can cause unpredictable results. Furthermore, problems can also occur when one CPU is modifying data that another CPU is reading. A protocol called MESI is employed on all Intel multi-processor configurations to ensure that each CPU is guaranteed to get the most recent copy of data even when other CPUs are currently using that data. MESI stands for modified, exclusive, shared, and invalid—the four possible data states that data can have when stored in a processor cache. One of these four states is assigned to every data element stored in each CPU cache. To support the MESI protocol, regular communication must occur between every CPU whenever data is loaded into a cache. On each processor data load into cache, the processor must broadcast to all other processors in the system to check their caches to see if they have the requested data. These broadcasts are called snoop cycles, and they must occur during every memory read or write operations. During the snoop cycle phase, each CPU in the SMP server checks its cache to see if it has the data that is being addressed by the requesting CPU. If the data is present in another CPU cache and the data has been modified, then that CPU must provide the data to the other requesting CPU. If the data is in some other CPU cache but it has not been modified, then that CPU must mark its data as shared or invalid, depending upon the type of operation that is requested on the front-side bus. If the operation is a write request, the CPU that is possessing the unmodified data must mark its data as invalid, indicating that the data can no longer be used. If the front-side bus request is a read operation, the data is marked as shared, indicating that its copy is read-only and cannot be modified without notifying the other CPUs. In this case, the CPU that generated the front-side bus read request will also mark its copy of the data in its cache as shared. Should either CPU then
Chapter 7. Chipset architecture
105
execute an instruction to modify the data (a write instruction), another front-side bus cycle occurs to inform the other CPUs to invalidate the data in any of their caches. At the completion of the snoop cycle for the write operation, the CPU updating the data marks the data as modified. The exclusive state is used to indicate that data is stored in only one cache. Data that is marked exclusive can be updated in the cache without a snoop broadcast to the other CPUs. This is possible because at the time the data was read from memory, no other CPUs indicated that they had ownership of the same data in their caches. So, the MESI state of the data was set to exclusive. Unless another CPU generated a front-side bus request for the data (in which case, the data would be marked as shared), it would stay exclusive. If the data were modified, the write operation that performed the update would cause the state of the data to be set to modified. Any subsequent requests for the modified data would be satisfied by the cache providing the modified data to the requesting CPU. The complete MESI protocol is quite complex and understanding all of its details is not necessary to appreciate its impact on SMP scalability. We simply introduce the protocol here so that you can be aware of the overhead that is required each time CPUs are added to an SMP server.
The MOESI protocol The AMD Opteron uses a slightly different version of the MESI protocol: MOESI. The MOESI protocol expands the MESI protocol with yet another cache line status flag namely the owner status (thus the O in MOESI). After the update of a cache line, the cache line is not written back to system memory but is flagged as an owner. When another CPU issues a read, it gets its data from the owner’s cache rather than from slower memory, thus improving memory performance in a multi-processor system. For more information about MOESI, see the Chip Architect article about the AMD64 processor at: http://chip-architect.com/news/2003_09_21_Detailed_Architecture_of_AMDs _64bit_Core.html#3.18
Large SMP configurations In general, the snoop overhead increases as the number of CPUs increases. Snoop overhead can also increase as the size of the caches increases. The larger the L2 cache, the greater the probability that snoop requests will hit data in another processor cache. These cycles delay the execution rate of the CPUs, because they must wait for the processor with ownership of the data to provide the data to the requesting CPU.
106
Tuning IBM System x Servers for Performance
Dramatic increases in front-side bus speed plus Hyper-Threading and multiple processor cores sharing the same front-side bus have caused issues that are related to snoop latency. When the number of CPU cores increases to eight or more, unless special optimization such as in the XA-64e chipset are made, CPU performance can bog down. Many of the early Pentium Pro eight-socket SMP systems performed more slowly with eight CPUs than with four processors. This was due primarily to the long snoop latencies that resulted from each CPU juggling shared and modified data. The snoop latency issue is exacerbated for many Intel SMP eight-socket configurations because these systems actually consist of two four-socket systems that are connected together by a specially designed dual-ported memory controller. This architecture is necessary because the Intel front-side bus protocol is limited to four processor sockets. To increase the number of CPUs beyond four, two or more independent front-side buses must somehow be connected together. The front-side bus to front-side bus connection had the potential to introduce overhead, which often meant the time to snoop the CPU caches on a remote front-side bus was much longer than the time to snoop the CPU caches on the local front-side bus. In general, this explains why there is a discontinuity in performance improvement from the fifth CPU compared to the gains that are obtained from the first to the fourth CPU. The increases in the number of processor threads which include the additional cores plus the Hyper-Threading has changed the traditional methodology for designing processors. New architectures are starting to incorporate multiple front side buses per memory controller. The penalty to snoop a processor on a second front-side bus heavily increases the front-side bus utilization. This penalty is why many architectures are starting to incorporate a snoop filter similar to the XA-64e chipset in their designs. The high front-side bus utilization explains why performance is optimal when splitting CPUs across two front-side busses, instead of fully populating a single front-side bus. Solutions to the performance problem are the use of the cache coherency filter or directory and “higher” levels of cache.
Chapter 7. Chipset architecture
107
Cache coherency filter One significant hardware optimization to enhance the performance of high-end systems is the cache coherency filter. Typically, one filter is used for each group of four processors. To think of this another way, each filter is used to track all the operations that occur on the front-side bus. The filter provides information about the addresses of all data that is stored in the caches of the CPUs on the respective front-side bus. These filters are used to store bits that indicate the presence of each cache line that is stored in all the caches of the processors. Whenever an address is snooped by a CPU, the memory controller looks up the address in the filter for the remote front-side bus (without an actual cycle to the remote front-side bus). If the remote filter responds with a hit, then and only then is the snoop cycle propagated to the remote front-side bus. If an address that is snooped is not present in the filter, then a snoop miss occurs, and the snoop completes quickly because it does not propagate to the remote bus. Remember, the CPU that requests the data that caused the snoop cycle might be waiting for the snoop cycle to complete. Furthermore, the front-side bus cannot be used by other CPUs during the snoop cycle, so snoop cycles must execute quickly to obtain CPU scalability. For more information, see 7.3.4, “XA-64e third generation chipset” on page 116.
7.2.5 Software scalability Adding processors improves server performance because software instruction execution can be shared among the additional processors. However, the addition of processors requires software to detect the additional CPUs and generates additional work in the form of threads or processes that execute on the additional processors. The operating system provides a platform that enables the capability of multiprocessing, but it is up to the application to generate the additional threads and processes to execute on all processors. This ability is referred to as application scalability. Faster server hardware means more parallelism (more processors, larger memory, larger disk arrays, additional PCI buses, and so on). The obvious case of software that does not scale is DOS. If you run DOS on a server with 8 CPUs and 64 GB of memory that is equipped with 250 15 K RPM disk arrays, you get about the same performance as though you have one CPU, one disk, and 640 KB of memory. Obviously, the server is not slow. The problem is that the software (in this case DOS) does not scale. This example is extreme, but it makes it easier to understand how software must actually evolve to take advantage of more powerful server hardware.
108
Tuning IBM System x Servers for Performance
Software scalability is a complex subject, one that most people do not consider until it is too late. Often people purchase new high-performance servers expecting huge performance gains with old applications, only to learn that the bottleneck is in the server application. In this case, there is little that they can do to efficiently use the new server until the application is modified. A scalable application makes use of greater amounts of memory, generates scalable I/O requests as the number of disks in a disk array increases, and will use multiple LAN adapters when a single LAN adapter limits bandwidth. In addition, a scalable application has to detect the number of installed processors and spawn additional threads as the number of processors increases to keep all processors busy. Hyper-Threading increases the number of logical processors and demands that the software spawn additional threads to run at maximum efficiency. However, some applications do not yet do this. This is why, in general, Hyper-Threading performs quite well with two-socket and four-socket, single core SMP systems, because many applications already generate sufficient threads to keep four physical/logical CPUs busy. However, at four-socket dual core, eight-socket, and 16-socket, the applications have to spawn even more threads to efficiently utilize Hyper-Threading or the additional cores. All of these things must be engineered into the server application and operating system. In general, the only applications that scale past four-socket are middle tier applications, database applications, and virtualization applications.
Multi-processing and server types Multi-processing has a direct relationship with the type of application server that is used. If the server is used as a file server, adding a processor to a file server does not improve performance significantly, while it can result in a very high performance gain for an application server. Multi-processing will not provide a linear improvement in processing power as additional processors are added. You might achieve a 70% to 80% performance increase from the second processor, but each additional processor will provide less and less performance increase as other system bottlenecks come into play, as illustrated in Figure 7-2 on page 110.
Chapter 7. Chipset architecture
109
Relative database Random transaction processing performance
Relative performance improvement
3 1.61x
2.8
2.5 2.44
2
1.4x 1.74
Memory doubled
1.5 1
1
No memory added
0.5 0
1 CPU 2 CPUs 4 CPU 4 CPU 6 GB memory 12 GB memory 12 GB memory 24 GB memory
Figure 7-2 Relative performance scaling from adding 3.0 GHz processors
Using this chart, another point should be made. Adding memory is critical to getting the most out of any processor upgrade. In this case, adding the third and fourth processors improves relative performance by 2.44, but if you also add memory, then the relative improvement is 2.8. The scalability of multi-processor systems also greatly varies if a NUMA or SMP design is used. Classic SMP designs scale very poorly beyond four-socket while NUMA systems show good scalability up to 32-socket (and even further on some very special NUMA systems)
7.3 Memory controller-based chipset Intel-based chipsets use an architecture where all processors reside connected to a front side bus. This front side bus allows for direct processor-to-processor as well as processor-to-memory communication. The memory controller handles communications that occur between the CPU, RAM, PCI Express controller, and the south bridge (I/O controller hub). The memory controller is the main aspect of a chipset that determines the number, speed, and type of CPU, as well as the number, speed, and type of main memory (RAM). This section describes some of the differences between different chipsets that are incorporated in servers in current server architectures.
110
Tuning IBM System x Servers for Performance
7.3.1 ServerWorks Grand Champion 4.0 HE and LE The ServerWorks Grand Champion 4.0 HE and LE chipsets are designed to work with the IA-32 platforms. These chipsets are targeted at servers that provide 2-way (LE) and 4-way (HE) SMP capabilities. This chipset is currently used in the following models: xSeries 235 xSeries x335 HS40 blade The Grand Champion HE chipset are used in servers to provide 4-way functionality while the Grand Champion LE chipset are used in rack dense and tower 2-way configurations. The Grand Champion 4.0 LE chipset incorporates the following features:
Two Intel Xeon processors up to 3.2 GHz 533 MHz front-side bus Up to 16 GB memory Support for PC2100 DDR memory (133 MHz) PCI-X slots for expansion
The LE chipset memory controllers support 2-way interleaved PC2100 DDR SDRAM at 266 MHz, which provides for a maximum bandwidth of 2.3 GBps. The Grand Champion 4.0 HE chipset incorporates the following features:
Four Intel Xeon MP CPUs up to 1.6 GHz 400 MHz front-side bus Up to 64 GB of memory Support for PC1600 DDR memory (100 MHz) PCI-X I/O slots for expansion
The HE chipset memory controller features support 4-way interleaved PC1600 DDR SDRAM at 100 MHz. This support ensures that memory access is kept synchronous with the 400 MHz front-side bus and provides a memory bandwidth of 6.4 GBps. Both of these memory controllers support Chipkill™ and mirrored memory configurations.
Chapter 7. Chipset architecture
111
7.3.2 Intel E7520 and E7525 The Intel E7520 and E7525 chipsets support the Intel EM64T 64-bit extensions as described in “64-bit extensions: AMD64 and Intel 64 Technology” on page 62 and PCI Express interface as described in 6.3, “PCI Express” on page 90. The E7520 (Intel code named Lindenhurst) is implemented in the following current IBM System x models:
xSeries 226 xSeries 236 xSeries 336 xSeries 346 System x3650 T HS20 blade Note: The E7525 (Intel code named Tumwater) is similar to the E7520 but is targeted at high-end workstations and low-end servers. It includes support for PCI Express x16 for video.
These chipsets have the following features:
112
Two Xeon processors, up to 3.8 GHz, 1 MB L2 cache Support for EM64T 800 MHz front-side bus Support for DDR2-400 memory Support for PCI-X slots Support for PCI Express slots (two x8)
Tuning IBM System x Servers for Performance
Figure 7-3 shows the block diagram of the System x236. CPU 1
400 MHz 2-way interleave
CPU 2
6.4 GBps 800 MHz
4x DDR2
E7520 Memory controller
4x DDR2
I/O Controller Hub
PCI bridge
5287
2x Gigabit Ethernet
Video Bus A
B
C
Bus D
Bus E
USB Legacy Mgmt
Ultra320 SCSI
32-bit 33 MHz PCI Express x8
64-bit 133 MHz 2x 64-bit 100 MHz
Figure 7-3 System x236 block diagram with two PCI Express slots
The CPUs are connected using a 200 MHz front-side bus but transmit data at a rate of 800 MHz using the quad-pumped design. This design means that the clock signals are interleaved and that data is supplied on the rising and falling edge of the clock signal, which results in an effective bandwidth of 6.4 Gbps across the front-side bus. The E7520 memory controller supports dual data paths to memory which run at 400 MHz and support 2-way interleaved PC-3200 DDR2 DIMMs. The memory throughput effectively matches the high performance speed of the Intel Xeon processors. A total of 16 GB of memory can be accessed at a transfer rate of 6.4 Gbps. Memory mirroring and spare memory are also supported in this system. Memory mirroring decreases the amount of total memory that is available but provides a RAID-1 redundancy for memory. RAID protection effectively keeps two copies of the data in separate banks of memory modules. A dual port Gigabit Ethernet adapter is built onboard and interfaces directly to the memory controller. The PCI subsystem is not used to provide Ethernet connectivity.
Chapter 7. Chipset architecture
113
One of the most significant enhancements of the memory controller is the addition of two PCI Express x8 connections. These adapters are attached directly to the memory controller, and the data does not need to transverse the PCI bridge. Note: The x346 does not support PCI Express as standard, but you can install an optional riser card to replace PCI-X slots 3 and 4 with PCI Express slots. For more information, see: http://www.intel.com/design/chipsets/E7520_E7320 http://www.intel.com/products/chipsets/E7525/
7.3.3 Intel 5000 chipset family The Intel 5000 chipset family exists as the follow-on to the E7520 chipset family and includes the following features: New generation of dual-core Intel Xeon 5000 series processors, including Woodcrest and Dempsey (for a detailed discussion, see 4.1.2, “Dual core Intel Xeon processors” on page 45). Intel Virtualization Technology (for a detailed discussion, see 5.2, “Virtualization hardware assists” on page 82). I/O Acceleration Technology (for a detailed discussion, see 10.3.2, “I/O Accelerator Technology” on page 267). Support for Fully Buffered DIMMs (for a detailed discussion, see 8.1.6, “Fully-buffered DIMMs” on page 137). Hyper-Threading and Enhanced Intel SpeedStep® Technology (for a detailed discussion, see “Hyper-Threading” on page 39). Intel 64 Technology (EM64T) (for a detailed discussion, see “64-bit computing” on page 61). The Intel 5000P Blackford is implemented in the IBM System x3400, x3500 and x3650, and the Intel 5000X chipset Greencreek is implemented in the IBM System x3550 model. The Intel 5000X Greencreek chipset differs from the Blackford chipset by its inclusion of a first generation snoop filter. The 5000 class chipset supports two processor sockets on dual, independent front-side buses that each operate at 266 MHz. The front-side buses are 64-bit, quad pumped, which allows for a total peak bandwidth of 17 GBps total or 8.5 GBps per front-side bus. Front-side bus speeds range from 1066 MHz to 1333 MHz.
114
Tuning IBM System x Servers for Performance
Figure 7-4 shows the block diagram for the System x3650 which includes the Intel 5000P Blackford chipset. Two processor and up to 12 Fully Buffered DIMMs are support with this chipset. The PCI-Express adapters are connected directly to the Blackford memory controller.
CPU
CPU 4-way interleaved Fully buffered DIMMs
PCI Express
Blackford Memory controller
ServeRAID SAS
HDD backplane External port
South bridge PCI Express x8 PCI Express x8 USB 2.0 PCI Express x4 Video PCI Express x4 Serial Gigabit Ethernet
Figure 7-4 System x3650 Block Diagram
For more information, see: http://www.intel.com/products/chipsets/5000P/ http://www.intel.com/products/chipsets/5000X/ http://www.intel.com/products/chipsets/5000V/
Chapter 7. Chipset architecture
115
7.3.4 XA-64e third generation chipset The following X3 Architecture servers employ the third generation IBM XA-64e chipset:
xSeries 260 xSeries 366 xSeries 460 System x3800 System x3850 System x3950
The IBM XA-64e chipsets supports 64-bit Intel Xeon processors which are compatible with 32-bit or 64-bit code. The XA-64e chipset includes the following components: Combined Memory and Cache controller: An integrated memory controller that improves response times and overall performance by reducing memory latency is code named Hurricane. Supports from one to four Xeon MP processors per node, including single-core Cranford or Potomac processors and dual-core Paxville or Tulsa Intel-based processors. PCI bridges: two Calgary PCI bridges. A XceL4v cache/snoop filter to lower congestion that occurs on the front-side bus. DDR2 memory support
116
Tuning IBM System x Servers for Performance
Figure 7-5 shows the block diagram of the x3850, showing the XA-64e chipset. CPU 1
CPU 2
667 MHz 5.33 GBps
DDR2
SMI2
DDR2
SMI2
DDR2
SMI2
DDR2
SMI2
Each: 667 MHz 5.33 GBps
PCI-X bridge 33 66
CPU 4
667 MHz 5.33 GBps
IBM XA-64e core chipset Scalability ports Each: 6.4 GBps (x460 and MXE-460 only)
Memory controller ("Hurricane")
6 GBps
Calgary
CPU 3
6 GBps
6 GBps
PCI-X bridge
266 266 MHz
266 MHz
Video RSA SL 1 USB 2.0
ServeRAID
South bridge
Adaptec SAS
EIDE
Gigabit Ethernet
K/M
2
3
4
5
6
HDD backplane Six PCI-X 2.0 slots: 64-bit 266 MHz
Serial
Figure 7-5 X3 Architecture System Diagram
The XA-64e chipset includes the following improvements: Support for Intel 64 Technology (EM64T), which allows for a physical address width of 40-bits or 1 TB theoretical maximum compared to 36-bit in XA-32. The single front-side bus was split to allow a 10.66 GBps front-side bus bandwidth versus 3.2 GBps bandwidth (3x gain). I/O controller chip support upgraded from PCI-X 133 MHz to PCI-X 2.0 266 MHz bandwidth which improves the RAS and bandwidth. Major focus on latency reduction, which allows for flat memory scaling similar to an SMP.
Chapter 7. Chipset architecture
117
400 MHz L4 cache chip removed and replaced with XceL4v to improve multinode scalability and snoop filter. Node controller chip merged with memory controller chip, which allowed for increases in main store bandwidth and reduction of store latencies. The physical address width of the XA-64e is 40 bits, meaning a theoretical maximum of 1 TB of memory. However, the x3950 limits the amount of memory to 512 GB (64 GB on each node in a 8-node configuration). This address width is implemented on the processor bus (front-side bus) and memory/scalability controller (Hurricane). The x3950 can support up to 32 dual core processors with Hyper-Threading enabled or 128 total processor threads. However, at the time of this redbook publication, the highest number of threads that an Intel or AMD compatible operating system can support is 64. So, full 8-node x3950 configurations need Hyper-Threading disabled for the software to function.
XceL4v Dynamic Server Cache The XceL4v dynamic server cache serves two purposes in the X3 Architecture servers: As a single, 4-way server (x3800, x3850, x3950), the XceL4v and its embedded DRAM (eDRAM) is used as a snoop filter to reduce traffic on the front-side bus. It stores a directory that consists of the tag and index of all processor cache lines to minimize snoop traffic on the dual front-side buses and minimize cache misses. When the x3950 is configured as a multi-node server, the eDRAM is used to store local and remote directory information that consists of a tag and an index. In addition, this technology allocates up to 256 MB of main memory dynamically in each node to cache remote node data. In an 8-node configuration, this means there will be 2 GB of main memory in use for a remote cache. With advances in chip design, IBM has now reduced the latency of main memory to below that of a local L4 cache in the earlier x445 system. In other words, the time it takes to access data directly from memory is almost as fast as accessing it from an L3 cache. As a result, on a four-socket single node system such as the x3800 or x3850, there is little or no need for either a L3 cache or L4 cache. By analysis of prior generation chipsets, IBM system designers realized that if a full L3 or L4 cache were instantiated, then the impact of cache misses would add significant overhead to overall memory access times. In most server applications with multiple users, the threads competing for an L3 cache generate a lower hit rate, and the latency of a full additional cache drops performance. The same applies to any L4 cache.
118
Tuning IBM System x Servers for Performance
As a result, there is no performance benefit in implementing a full L3 or L4 cache in the single node System x3800 or x3850. For these reasons, in a single node system, the XceL4v acts as a snoop filter where only a tag and an index are cached and not data for the four-socket single node systems. The directory consisting of the tag and index is 9-way set associative for single node systems, which was determined through testing to produce the most optimal results for server workloads. Multi-node systems such as the 8-socket, 2-node x3950 incorporate the eDRAM of the XceL4v dynamic server cache to store address information for other nodes in addition to the processors on the local node. Therefore, the XceL4v dynamic server cache exists as a snoop filter caching both directory information for the local processors and the directory information from scalability traffic. In addition, main memory is allocated in order to cache scalability data from a remote node. Figure 7-6 on page 120 indicates the performance increases that are obtained by the inclusion of the XceL4v dynamic cache or snoop filter with a variety of different workloads. A lower front-side bus utilization performs better because it allows transactions to occur in the same amount of time. (The characteristics that are required for snoop traffic are described in detail in 7.2.4, “The MESI protocol” on page 105.) The addition of the XceL4v dynamic cache decreases front-side bus utilization an average of 47% over systems that do not incorporate a snoop filter. This effectively doubles the available 667 MHz front-side bus bandwidth over a non-snoop filter based high end chipset.
Chapter 7. Chipset architecture
119
IBM X3 Architecture System Traffic Reduction = Performance Increase Performance increases achieved by reduction of traffic between system resources (such as processors, memory, and I/O)
Traffic Without X3
60,000,000 Front-Side Bus Transaction (10 sec sample)
Traffic With X3 44% Reduction 50,000,000 46% Reduction
51% Reduction 40,000,000
48% Reduction
30,000,000
20,000,000
10,000,000
0 Complex Mixed Workload
Messaging
Large Object Transfer (High Netw ork Utilization)
Cached Workload
Scenario
Figure 7-6 X3 Architecture snoop filter performance gain
For more information about the XA-64e Architecture, see Planning and Installing the IBM Eserver X3 Architecture Servers, SG24-6797, which is available at: http://www.redbooks.ibm.com/abstracts/sg246797.html You can also consult the white paper, IBM X3 Architecture: Application Server Performance Gains, which is available at: http://www.ibm.com/support/docview.wss?uid=tss1wp100726
120
Tuning IBM System x Servers for Performance
7.3.5 Intel E8500 Chipset The Intel E8500 chipset, code named Twin Castle is used in many of our competitors’ high-end systems. The E8500 is Intel’s high-end chipset that was created for Intel EM64T MP-based, single-core Cranford and Potomac as well as dual-core Paxville and Tulsa processors. This chipset is a continuation or upgrade of the previous line of high-end Intel chipset and has the following features: Two front-side buses per node similar to the X3-64e instead of a single front side bus per node. The front-side bus has a bandwidth up to 800 MHz. Memory support for DDR-266, DDR-333, and DDR2-400. EM64T and support for single-core Cranford or Potomac and dual-core Paxville and Tulsa processors. PCI-E Support 3x8 and 1x4 lanes. The E8500 does not incorporate a snoop filter. Figure 7-6 on page 120 shows the higher front-side bus utilization of a high-end chipset that does not incorporate a snoop filter such as the E8500. Processor commands sourced from one front-side bus generates traffic on the second front-side bus to ensure that those other processors do not have the requested data in their cache. This increase in traffic drives up the front-side bus utilization and slows down the overall performance of the system. Despite being released in March 2005, there have not been many performance benchmarks that demonstrate the performance of the E8500 “Twin Castle” chipset.
Chapter 7. Chipset architecture
121
The TPC-C benchmark is used generally to benchmark high-end servers by running an online-transaction processing workload. Figure 7-7 shows the performance difference between the only published four-socket Intel E8500 chipset verses four-socket X3-64e systems running the TPC-C benchmark. All three benchmark configurations used 3.0 GHz Paxville processors with 2x2 MB L2 caches and 64 GB of memory in the server. TPC-C 4-socket Chipset Perform ance 300,000
TPM-C Performance
250,000 200,000 23.75%
150,000 100,000
17%
50,000 0 Intel E8500 Chips et
X3-64e Chipset (xSeries 366)
X3-64e Chipset (xSeries 460)
Note: TPC-C results referenced are Fijitsu Siemens Primergy TX600 S3 running SQL Server 2000, 188,761 tpmC, $5.70 $/tpmC, available 4/27/06, IBM x366 running DB2 UDB 8.2, 221,017 tpmC, $8.27 $/tpmC, available 3/31/06, IBM x460 running DB2 UDB 8.2, 273,520 tpmC, $4.66 $/tpmC, available 5/01/06. Results as of 6/20/06.
Figure 7-7 Four-socket TPC-C chipset performance
The results illustrate that the single node X3-64e chipset (xSeries 366) outperforms the Intel E8500 chipset by 17% for an online transaction processing workload despite the fact that the Intel E8500 chipset was running a 800 MHz front side bus compared to the 667 MHz front side bus of the X3-64e configuration. The X3-64e chipset was also configured with 4 CPUs configured as 2-nodes with two processors per node and a total of 128 GB memory which is represented by the xSeries 460 data point. The x460 result illustrates the scalability of multiple nodes by keeping the number of processors constant. For more information, see: http://www.intel.com/products/chipsets/e8500/
122
Tuning IBM System x Servers for Performance
7.4 PCI bridge-based chipsets AMD Opteron processors do not use the typical shared front-side bus that is connected to a memory controller used in Intel-based servers. Each Opteron processor has its own integrated memory controller and pins on the processor chip to directly connect to a memory bus. So, in Opteron, processor and memory controller logic are integrated into the same piece of silicon, eliminating the need for a separate memory controller part. Hardware vendors simply add a memory bus and memory DIMMs and they have the core CPU and memory interface. To keep data coherent between multiple Opteron processors, AMD introduced a new system bus architecture called HyperTransport. Three HyperTransport links are available on each Opteron processor, two used for CPU-CPU connectivity and one used for I/O. The two HyperTransport links used for CPUs enable the direct connection of two processors and the indirect connection of four or more processors. IBM System x and BladeCenter servers that have this type of architecture include the following:
IBM e326m System x3455 System x3755 BladeCenter JS20/JS21 BladeCenter LS20/LS21
Chapter 7. Chipset architecture
123
With a four-processor configuration, the processors are placed at the corners of a square, with each line that makes up the square representing a HyperTransport connection between the processors. See Figure 7-8. With this design, whenever two processors that on the same side of the square share data, the information passes directly over the HyperTransport interconnect to the other processor. When this remote access occurs, it is called a single hop remote memory access and is slower than a local memory access.
CPU & memory
CPU & memory
Two hop remote memory access
CPU & memory
Single hop remote memory access
CPU & memory
Figure 7-8 Remote memory access
However, when two processors on diagonal corners of the square share data or instructions, the information must travel through an additional processor connection before arriving at the diagonal processor. This extra hop adds some additional overhead and is referred to as a two hop remote access.
124
Tuning IBM System x Servers for Performance
In systems such as the System x3755, when the server is configured with just three processors, a passthru card can be installed in place of the fourth processor to reduce the two-hop access to just a single hop, as shown in Figure 7-9. For more information, see “IBM CPU passthru card” on page 56.
Without a passthru card, cards 1and 3 are two hops away
Card 3
Card 2
Card 1
Passthru With a passthru card, cards 1 & 3 are connected
Card 3
Card 2
Card 1
Figure 7-9 The benefit of the passthru card for three-way configurations
The third port of the HyperTransport link is not used to interconnect the diagonal processors because it must be used for connection to a PCI I/O bridge which connects such devices as PCI slots, network, disk storage, mouse, keyboard, video, and so forth. Officially, Opteron processors support up to eight CPUs within a single system. However, the latency to include all eight sockets and the additional hops in a single architecture would add little to no performance gains over a four socket system. The remote memory access latency of a processor accessing another processor’s memory space makes the Opteron configuration a NUMA design (refer to 7.2.3, “NUMA” on page 101). NUMA means that every processor has memory that is closer and thus more rapidly accessible and memory that is remote and slower that must be accessed through another Opteron processor. AMD refers to its Opteron architecture as sufficiently uniform memory organization (SUMO) rather than NUMA. From an architectural standpoint, it still is a NUMA architecture but the HyperTransport link is fast enough to run software written for SMP systems without very significant performance penalties. Current operating systems such as the latest versions of Linux and Windows 2003 SP1 support NUMA and make attempts to minimize remote memory transactions. However, in practice, the percentage of remote memory accesses is largely determined by application behavior and by how data is manipulated by users of the application.
Chapter 7. Chipset architecture
125
Figure 7-10 shows the Opteron architecture with the integrated memory controller and the HyperTransport connectors. Connections to other processors or I/O devices
HyperTransport
L1 Data cache Processor core
L2 cache L1 Instruction cache
Integrated DDR memory controller
Memory Figure 7-10 CPU architecture of the Opteron CPU with an integrated memory controller
HyperTransport The HyperTransport architecture was initially developed by AMD but is now managed by an open consortium of several big IT companies such as AMD, Apple, Cisco, Broadcom, ATI, IBM, and many others. HyperTransport is an open standard for a high-speed, point-to-point link system that can be used for connecting a variety of chips. The HyperTransport technology is used in devices such as network devices, graphics cards or, as in the case of the AMD Opteron, as a high-speed interconnect for processors. The HyperTransport technology used for interconnecting Opteron processors is currently implemented at a speed of 1000 MHz with a bidirectional bandwidth of 4.0 GBps each way that leads to a peak full-duplex capacity of 8.0 GB per second per link. Current Opteron processors incorporate three HyperTransport links which enables a peak bandwidth of 24 GBps per processor.
126
Tuning IBM System x Servers for Performance
You can find more information about the HyperTransport and the HyperTransport Consortium at: http://www.hypertransport.org/
Chapter 7. Chipset architecture
127
128
Tuning IBM System x Servers for Performance
8
Chapter 8.
Memory subsystem Insufficient memory might often be the reason behind poor server performance. As the amount of memory that running programs and their data uses approaches the total available physical memory that is installed on the machine, a server’s virtual memory handler increases the amount of data that is paged in and out of memory, to and from the paging area on disk, with a disastrous effect on performance. Fortunately, the memory subsystem is usually one of the easiest areas of the entire system to upgrade. Over the years, memory capacity demands have increased because server operating systems and server application code have grown. Additionally, user content has evolved from simple character-based data to more expansive rich media such as audio and video. Applications and data that were stored on large high-end computers previously have migrated down to Intel class servers, placing additional demands on memory capacity. Trends indicate an increasing demand for server memory capacity well into the future. When comparing the speed of CPU cache, main memory, and online disk storage media, we can see that cache is the fastest but also the most expensive per megabyte, while disk is the cheapest per megabyte but with orders of magnitude slower than cache memory. The use of main memory is a good compromise between speed and price. The L1, L2, and L3 CPU caches (and the IBM XceL4v cache) bridge the performance gap between processor and memory. Memory bridges the performance gap between the caches and disk.
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
129
Network-attached users usually access unique data objects on the server. Each of these data objects requires memory for storage. Furthermore, each task that a network user requests the server to perform requires many thousands of instructions to be executed. Network users do not usually require the same tasks to be performed by the server at the same time. Thus, memory is also needed to hold the programs and DLLs that are required by each user. Server memory requirements increase as the number of users grows. A server that does not have sufficient memory to meet the requirements of all active users will attempt to expand storage onto the disk drive. When this occurs, server performance suffers because disk accesses are significantly slower than memory accesses. There are three main memory technologies in current System x servers: DDR DDR2 Fully buffered DIMMs This chapter describes each of these memory types and covers the following topics:
130
8.1, “Memory technology” on page 131 8.2, “Specifying memory performance” on page 144 8.3, “SMP and NUMA architectures” on page 147 8.4, “The 32-bit 4 GB memory limit” on page 151 8.5, “64-bit memory addressing” on page 155 8.6, “Advanced ECC memory (Chipkill)” on page 156 8.7, “Memory mirroring” on page 157 8.8, “X3 architecture servers” on page 158 8.9, “IBM Xcelerated Memory Technology” on page 164 8.10, “BIOS levels and DIMM placement” on page 165 8.11, “Memory rules of thumb” on page 165
Tuning IBM System x Servers for Performance
8.1 Memory technology This section introduces some of the key terminology and technology that are related to memory. Topics that we discuss here are:
8.1.1, “DIMMs and DRAMs” on page 131 8.1.2, “Ranks” on page 133 8.1.3, “SDRAM” on page 134 8.1.4, “Registered and unbuffered DIMMs” on page 134 8.1.5, “Double Data Rate memory, DDR and DDR2” on page 135 8.1.6, “Fully-buffered DIMMs” on page 137 8.1.7, “DIMM nomenclature” on page 141 8.1.8, “DIMMs layout” on page 142 8.1.9, “Memory interleaving” on page 143
8.1.1 DIMMs and DRAMs Memory in servers is implemented in the form of Dual Inline Memory Modules (DIMMs). DIMMs contain a number of chips, known as Synchronous Dynamic RAM (SDRAM or just DRAM) chips. The number of chips implemented on the DIMM depends on the total capacity of the DIMM and whether the DIMM has error checking and correcting (ECC) functions. Without ECC, a DIMM typically has 8 or 16 SDRAM chips. With ECC, there are typically 9 or 18 chips. Figure 8-1 is a photo of an ECC DIMM, with 9 SDRAM chips on each side.
A DRAM chip on a DIMM. The last chip on the right is the ECC chip. There are 8 DRAM + 1 ECC chip on this side of the DIMM. Figure 8-1 DRAM chips on a DIMM
The capacity of each DRAM is a number of “words” where each word can be 4 bits (“x4”), 8 bits (“x8”) and, starting to become prevalent, 16 bits in length (“x16”). The word length is usually written as x4 for 4 bits, and so on. The number of words in the DRAM is sometimes written on the label of the DIMM, such as
Chapter 8. Memory subsystem
131
128M, meaning that each DRAM has 128 million (actually 128 x 10243 ) words. Figure 8-2 shows an example. Note: The word length (x4 or x8) is normally not printed on the label, however the DIMM manufacturer’s Web site might list such specifications. It can also be calculated: (DIMM capacity in MB) / (Number of non-ECC DRAMs) * 8 / (M value) So for the 1 GB DIMM in Figure 8-2, 1024 MB / 8 * 8 / 128 = 8 bit word length.
Capacity of each DRAM in words
Data width of DIMM: 64 bits = non-ECC 72 bits = ECC
Figure 8-2 DRAM capacity as printed on a PC3200 (400 MHz) DDR DIMM
The sum of the capacities of the DRAM chips (minus any used for ECC functions if any), equals the capacity of the DIMM. Using the previous example, the DRAMs in Figure 8-2 are 8-bits wide, so: 8 x 128M = 1024 Mbits = 128 MB per DRAM 128 MB x 8 DRAM chips = 1024 MB or 1 GB of memory
132
Tuning IBM System x Servers for Performance
8.1.2 Ranks A rank is a set of DRAM chips on a DIMM that provides eight bytes (64 bits) of data. DIMMs are typically configured as either single-rank or double-rank devices but four-rank devices are becoming more prevalent. Using x4 DRAM devices, a rank of memory is composed of 64 / 4 = 16 DRAMs. Similarly, using x8 DRAM devices, a rank is composed of only 64 / 8 = 8 DRAMs. A DIMM can contain 1 or 2 ranks. The DRAM devices that make up a rank are often (but not always) mounted on one side of the DIMM, so a single-rank DIMMs can also be referred to as a single-sided DIMM. Likewise a double-ranked DIMM can be referred to as a double-sided DIMM. Some servers do not support more than eight ranks. Therefore, when more than four double-ranked DIMMs are used, the server’s BIOS returns a memory configuration error. The workaround is to use single-rank DIMMs (same capacity). In that case, it is possible to use up to eight DIMMs. Note: Some servers do not allow the mixing of DIMMs with different numbers of ranks. Other systems do support mixing, but require DIMMs be placed in certain order. Other systems again do allow combinations of DIMMs with different ranks. Single-rank DIMMs and the double-ranked DIMMs are identified as follows in manufacturer’s technical sheets, depending on the model: x8SR = x8 single-ranked modules - have five DRAMs on the front and four DRAMs on the back with empty spots in between the DRAMs. x8DR = x8 double-ranked modules - have nine DRAMs on each side for a total of 18 (no empty slots) x4SR = x4 single-ranked modules - have nine DRAMs on each side for a total of 18 and look similar to x8 double-ranked x4DR = x4 double-ranked modules - have 18 DRAMs on each side for a total of 36. The rank of a DIMM also impacts how many failures a DIMM can tolerate using redundant bit steering. See “Memory ProteXion: Redundant bit steering” on page 163 for details.
Chapter 8. Memory subsystem
133
8.1.3 SDRAM Synchronous Dynamic Random Access Memory (SDRAM) is used commonly in servers today, and this memory type continues to evolve to keep pace with modern processors. SDRAM enables fast, continuous bursting of sequential memory addresses. After the first address is supplied, the SDRAM itself increments an address pointer and readies the next memory location that is accessed. The SDRAM continues bursting until the predetermined length of data has been accessed. The SDRAM supplies and uses a synchronous clock to clock out data from the SDRAM chips. The address generator logic of the SDRAM module also uses the system-supplied clock to increment the address counter to point to the next address.
8.1.4 Registered and unbuffered DIMMs There are two types of SDRAMs currently on the market: registered and unbuffered. Only registered SDRAM are now used in System x servers, however. Registered and unbuffered cannot be mixed together in a server. With unbuffered DIMMs, the memory controller communicates directly with the DRAMs, giving them a slight performance advantage over registered DIMMs. The disadvantage of unbuffered DIMMs is that they have a limited drive capability, which means that the number of DIMMs that can be connected together on the same bus remains small, due to electrical loading. Unbuffered DIMMs can manage one operation at a time. In contrast, registered DIMMs use registers to isolate the memory controller from the DRAMs, which leads to a lighter electrical load. Therefore, more DIMMs can be interconnected and larger memory capacity is possible. The register does, however, typically impose a clock or more of delay, meaning that registered DIMMs often have longer access times than their unbuffered counterparts. The registered DIMMs are able to mange operations in parallel. These differences mean that fewer unbuffered DIMMs are typically supported in a system than for a design using registered DIMMs. While this might not be a problem for desktop systems, servers now need large amounts of memory and use registered DIMMs.
134
Tuning IBM System x Servers for Performance
8.1.5 Double Data Rate memory, DDR and DDR2 Data transfers made to and from an SDRAM DIMM use a synchronous clock signal to establish timing. For example, SDRAM memory transfers data whenever the clock signal makes a transition from a logic low level to a logic high level. Faster clock speeds mean faster data transfer from the DIMM into the memory controller (and finally to the processor) or PCI adapters. However, electromagnetic effects induce noise which limits how fast signals can be cycled across the memory bus. Double Data Rate (DDR) memory techniques increase the data rate by transferring data on both the rising edge and the falling edge of the clock signal. DDR DIMMs use a 2x prefetch scheme so that two sets of 64-bit data are referenced simultaneously. Logic on the DIMM multiplexes the two 64-bit results (plus ECC bits) to appear on each of the rising and falling edges of the clock signal. Thus, two data transfers can be performed during one clock period. DDR2 is the new generation of DDR technology. The primary benefit is the potential for faster throughput. Currently, DDR2 operates at data transfer rates starting at 400 MHz and can reach 800 MHz. 1600 MHz support is expected in the future. DDR2 also improves the power consumption of the DIMM because it works on a lower voltage. DDR operates at a range of 2.5 V to 2.8 V, while DDR2 only requires 1.8 V (SDRAM DIMMs operate at 3.3 V). Figure 8-3 shows a standard DDR2 DIMM from the top and from the bottom.
Figure 8-3 A standard DDR2 DIMM (top) and small form-factor DDR2 DIMM (bottom)
Chapter 8. Memory subsystem
135
DDR2 consumes less power than DDR and offers a higher range of throughputs because it has half the speed of the memory core (thereby reducing power consumption) but offsets that by doubling the number of prefetches from the memory core to the I/O buffers (from 2 to 4), as shown in Figure 8-4. 200 MHz FSB
400 MHz I/O buffers 200 MHz
200 MHz FSB
400 MHz I/O buffers 200 MHz Double the number of prefetches
Memory core 200 MHz
Memory core 100 MHz
DDR at 400 MHz
DDR2 at 400 MHz
Half the speed of the memory core
Figure 8-4 Comparing DDR and DDR2 at the same external frequency
The lower frequency at the memory core means less power consumption and the ability to increase data density (and therefore capacity) and increase speeds as manufacturing technology improves. Note: The pin count has also changed. Thus, the two standards are not compatible. DDR has 184 pins and DDR2 has 240. At the time of the writing, DDR2 DIMMs are replacing DDR in AMD processor-based System x servers, and FB-DIMMs are replacing the current DDR2 on Intel based System x servers.
DDR2 performance As shown in Figure 8-4, when comparing DDR and DDR2 at the same external frequency (400 MHz dual-edge), the throughput is the same. In addition, because the internal core frequency of DDR2 is half that of DDR, there is more scope to increase frequencies and, therefore, increase the bandwidth of DDR2. However, the lower memory core frequency means longer latency time (that is, the time it takes to set up the request for data transfer).
136
Tuning IBM System x Servers for Performance
The end result of this is that at the DDR2 lower frequency of 400 MHz, which is the DDR upper frequency, the two technologies offer equal throughput, but the latency of DDR2 is worse. However, because DDR2 frequencies can increase (667 MHz is available) and DDR has reached its limit, throughput has increased and latencies have equalized.
8.1.6 Fully-buffered DIMMs As CPU speeds increase, memory access must keep up so as to reduce the potential for bottlenecks in the memory subsystem. With the DDR2 parallel memory bus design, all DIMMs on the one channel are connected to the memory controller. The problem is that as the speed of the memory channel increases, the number of DIMMs that can be connected decreases due to electrical loading. One solution is to add more channels, but that requires a significantly more complex circuit board design and larger board surface area for the additional wires. The fully-buffered DIMM (FB-DIMM) technology replaces the shared parallel memory channel that is used in traditional DDR and DDR2 memory controllers and uses a serial connection to each DIMM on the channel. As shown in Figure 8-5, the first DIMM in the channel is connected to the memory controller. Subsequent DIMMs on the channel connect to the one before it. The interface at each DIMM is a buffer known as the Advanced Memory Buffer (AMB). DDR2 memory DIMMs DDR2: Stub-bus topology
DIMM
DIMM
DIMM
DIMM
DDR memory controller
Fully buffered memory DIMMs FB-DIMM: Serial point-to-point links topology
DIMM
DIMM
DIMM
DIMM
Buffer
Buffer
Buffer
Buffer
FB-DIMM memory controller
Figure 8-5 Comparing the DDR stub-bus topology with FB-DIMM serial topology
Chapter 8. Memory subsystem
137
This serial interface results in fewer connections to the DIMMs (approximately 69 per channel) and less complex wiring. These links are relatively similar to the ones used for PCI-E, SATA, or SAS technologies. The interface between the buffer and DRAM chips is the same as with DDR2 DIMMs. The DRAM chips are also the same as DDR2 DIMMs. With this serial point-to-point connectivity, this is a built-in latency associated with any memory request. In addition, the design of FB-DIMM is such that even if the request is fulfilled by the first DIMM nearest to the memory controller, the address request must still travel the full length of the serial bus. As a consequence, the more DIMMs per channel that you have, the longer the latency Figure 8-6 shows the architecture of an FB-DIMM implementation, showing connections and placement of the buffer and DRAM chips. Standard DDR2 interface between buffer and DRAMs
North 10 bits FB-DIMM memory controller South 14 bits New serial interface between controller and buffer - up to six channels Industry standard DRAMs
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
Advanced Memory Buffer
Advanced Memory Buffer
Advanced Memory Buffer
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
DRAM
Up to eight DIMMs per channel
DRAM
Figure 8-6 Fully buffered DIMMs architecture
Note: FB-DIMMs are not the next generation of DRAM. It is a new way of accessing the same DDR2 DRAMs from a new memory controller.
138
Tuning IBM System x Servers for Performance
With the FB-DIMMs, the density can then increase without generating errors on data access to allow much more capacity scaling and a much larger memory bandwidth. An FB-DIMM memory controller can support up to 6 channels with up to 8 DIMMs per channel and single or dual-rank DIMMs. In addition, power consumption for an FB-DIMM is now 1.2 V versus 1.8 V for DDR2 DIMMs. Figure 8-7 shows an FB-DIMM.
FB-DIMM
DDR2
Figure 8-7 An FB-DIMM has the same connector as a DDR2 DIMM but a different key
Basically, FB-DIMMs allow for greater memory capacity in a server.
The Advanced Memory Buffer An FB-DIMM uses a buffer known as the Advanced Memory Buffer (AMB), as shown in Figure 8-8. The AMB is a memory interface that connects an array of DRAM chips to the memory controller. The AMB is responsible for handling FB-DIMM channel and memory requests to and from the local FB-DIMM and for forwarding requests to other AMBs in other FB-DIMMs on the channel.
Figure 8-8 The Advanced Memory Buffer on an FB-DIMM
Chapter 8. Memory subsystem
139
The functions that the AMB performs includes the following: Channel initialization to align the clocks and to verify channel connectivity. It is a synchronization of all the DIMMs on a channel so that they are all communicating at the same time. Support the forwarding of southbound frames (writing to memory) and northbound frames (reading from memory), servicing requests directed to a specific FB-DIMMs AMB and merging the return data into the northbound frames. Detect errors on the channel and report them to the memory controller. Act as a DRAM memory buffer for all read, write, and configuration accesses addressed to a specific FB-DIMMs AMB. Provide a read and write buffer FIFO. Support an SMBus protocol interface for access to the AMB configuration registers. Provide a register interface for the thermal sensor and status indicator. Function as a repeater to extend the maximum length of FB-DIMM Links.
FB-DIMM performance With a less complicated circuit board design and lower power consumption, server designers can now create memory controllers with more channels. The use of more memory channels results in better throughput. The use of a serial connection adds latency to memory access, but the greater throughput offered by FB-DIMM results in lower average latency when under load, thereby improving performance, as shown in Figure 8-9. Figure 8-9 shows us that with more channels, you can reach higher throughput. At low throughput levels, the latency of the serial link is significant, however because that latency remains constant regardless of the load, FB-DIMM performance is significantly better than DDR2 as throughput increases.
140
Tuning IBM System x Servers for Performance
Memory read latency
At low data rates, serialization adds latency compared to stub-bus architecture
DDR2-800
FB-DIMMs
More channels allow more throughput
2 DDR2-800 channels, 2 DIMMs per channel (4 total), 2 ranks/channel 4 FB-DIMM 800 MHz channels, 1 DIMM per channel (4 total), 1 rank/channel
Latency remains constant as throughput
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19
Theoretical peak memory throughput (GBps) Figure 8-9 Latency versus throughput for DDR2 and FB-DIMMs
8.1.7 DIMM nomenclature The speed of a memory DIMM is indicated by PC value, both for DDR and DDR2 DIMMs. The tables in this section list the nomenclature and the bus speed, transfer speed, and peak throughput. With DDR and DDR2, because the SDRAM transfers data on both the falling and the rising edges of the clock signal, transfers speeds are double the memory bus speed. Table 8-1 summarizes these values for DDR memory. Table 8-1 DDR memory implementations
DDR type
Bus speed
DDR transfers
Peak throughput
PC1600 (PC200)
100 MHz
200 MHz
1.6 GBps
PC2100 (PC266)
133 MHz
266 MHz
2.1 GBps
PC2700 (PC333)
167 MHz
333 MHz
2.7 GBps
PC3200 (PC400)
200 MHz
400 MHz
3.2 GBps
Chapter 8. Memory subsystem
141
Table 8-2 lists the current DDR2 memory implementations. Table 8-2 DDR2 memory implementations
DDR2 type
Bus speed
DDR transfers
Peak throughput
PC2-3200
200 MHz
400 MHz
3.2 GBps
PC2-4300
266 MHz
533 MHz
4.3 GBps
PC2-5300
333 MHz
667 MHz
5.3 GBps
PC2-6400
200 MHz
800 MHz
6.4 GBps
Because FB-DIMMs use DDR2 SDRAM, no specific name or performance changes have been introduced. You can find more detailed SDRAM specification information at: http://developer.intel.com/technology/memory/
8.1.8 DIMMs layout In general, the DIMM location within the system’s DIMM sockets does not affect server performance, although a particular server might require specific DIMM layouts for other reasons (such as initial detection of installed memory). In general, optimal performance is obtained when populating all DIMM slots with equivalent memory size DIMMs. This rule is also the case with modern DIMM technology such as DDR2 and FB-DIMM. Using the maximum number of the same-capacity DIMMs allows the memory controller to maintain the maximum number of open memory pages. Doing so reduces memory latency and gives the memory controller the option to enable address bit permuting (sometimes called symmetric mode or enhanced memory mapping), which reorders memory addresses to reduce random and sequential memory read/write latencies significantly. The significance of this discussion is that memory performance is highly dependent upon not just whether the data is in cache or main memory, but also how the access patterns appear to the memory controller. The access pattern will strongly affect how the memory controller reorders or combines memory requests, whether successive requests hit the same page, and so forth. Memory performance is affected by a number of complex factors. Those factors can include the choice of architecture, processor frequency, memory frequency, the number of DIMMs in the system, and whether the system is set up as a NUMA or an SMP system.
142
Tuning IBM System x Servers for Performance
8.1.9 Memory interleaving Interleaving is a technique that is often used to organize DIMMs on the motherboard of a server in order to improve memory transfer performance. The technique can be implemented within a single cache line access or across multiple cache lines to improve total memory bandwidth. When two DIMMs are grouped together and accessed concurrently to respond to a single cache line request, the interleave is defined as two-way; when four DIMMs are grouped together and accessed concurrently for a single cache line, the interleave is four-way. Interleaving improves memory performance because each DIMM in the interleave is given its memory address at the same time. Each DIMM begins the access while the memory controller waits for the first access latency time to expire. Then, after the first access latency has expired, all DIMMs in the interleave are ready to transfer multiple 64-bit objects in parallel, without delay (the front side bus is still 64 bits wide on the Xeon MP). Interleaving requires a 128-bit memory bus for two-way interleaving and a 256-bit memory bus for four-way interleaving. Systems designed around the Pentium 4 processor microarchitecture (Pentium 4, Xeon DP, and Xeon MP) usually employ a minimum of two-way interleaving using DDR DIMMs. In these systems, two-way interleaving is usually configured using two banks of DDR memory DIMMs. You can also set up four-way interleaving with four banks of DDR DIMMs. In either case, the memory subsystem is capable of transferring 64 bits of data at the full front-side bus speed of 400 MHz. The important points to remember are: Interleaving improves performance. If interleaving is optional, then we recommend you implement it to maximize your server investment by installing the appropriate number of DIMMs per the server documentation. DDR and DDR2 have similar interleaving characteristics. A single DDR or DDR2 DIMM transfers 64 bits at 200 MHz and therefore two-way DDR interleaving is required to match the 400 MHz transfer rate of the front-side bus.
Chapter 8. Memory subsystem
143
8.2 Specifying memory performance Performance with memory can be simplified into two main aspects: bandwidth and latency.
8.2.1 Bandwidth Basically, the more memory bandwidth you have, the better system performance will be, because data can be transferred faster. Bandwidth can be compared as a highway: the more lines you have, the more traffic you can handle. The memory DIMMs are connected to the memory controller through memory channels, and the memory bandwidth for a system is calculated by multiplying the number of bytes of a channel from a memory controller by the number of channels, and then multiplied by the frequency of the memory (or front side bus). For example, if a processor is able to support up to 400 MHz (DDR-400) registered ECC memory and has two 8-byte channels from the memory controller to access the memory, then the memory bandwidth of the system will be 8 bytes*2*400 MHz, or 6.4 GBps. If the memory used is DDR-333, the memory bandwidth will be 5.3 GBps. Models that use 533 and 667 MHz memory have a theoretical bandwidth of 4.3 GBps and 5.3 GBps per channel. Tip: The theoretical memory bandwidth does not depend on the memory technology (DDR-1 or DDR-2), but on the memory frequency, on the number of channels and on the size of these channels. If the request cannot be satisfied in cache, it is forwarded on through various mechanisms to the memory controller (for example, north bridge for Xeon or MCT for Opteron). The memory controller can hold incoming read or write requests in a queue while it completes other requests that are in progress. As new requests come in, they are checked against existing requests in the queue. If the new requests are related to requests currently in the queue, they can be combined or reordered to save time. Processors are designed to access memory using a structure called a cache line. Currently, a cache line is 64-byte wide, organized as 8 units of 64 bits (8 bytes). We discuss cache lines more in 4.3.2, “Cache associativity” on page 67.
8.2.2 Latency The performance of memory access is usually described by listing the number of front-side bus clock cycles that are necessary for each of the 64-bit transfers needed to fill a cache line. Cache lines are multiplexed to increase performance,
144
Tuning IBM System x Servers for Performance
and the addresses are divided into row addresses and column addresses. A row address is the upper half of the address (that is, the upper 32 bits of a 64-bit address), while a column address is the lower half of the address. The row address must be set first, then the column address. When the memory controller is ready to issue a read or write request, the address lines are set, and the command is issued to the DIMMs. When two requests have different column addresses but use the same row address, they are said to occur in the same page. When multiple requests to the same page occur together, the memory controller can set the column address once, and then change the row address as needed for each reference. The page can be left open until the it’s no longer needed. Or it can be closed after the request is issued. We talk about respectively page open or page closed policy. The act of changing a column address is referred to as Column Address Select or
CAS. There are actually four common access times: CAS: Column Address Select RAS to CAS: delay between row access and column access RAS: Row Address Strobe Sometimes, these numbers are expressed as x-y-y by manufacturers. These numbers are expressed in clocks and might be interpreted as wait times or latency. Therefore, the less these numbers are the better, because access times imply data access latency. CAS Latency (CL) measures the number of memory clocks that elapse between the time a memory controller sets the column address to request a line of data and when the DIMM is able to respond with that data. Even if other latencies are specified by memory manufacturers, CL is the most commonly used when talking about latency. If you look at the sticker on a DIMM (Figure 8-10), it might list the CL value for that particular device.
Chapter 8. Memory subsystem
145
CAS Latency value - CL3
Figure 8-10 CAS Latency value as printed on a PC3200 (400 MHz) DDR DIMM
CL values of 2.5 or 3.0 are typical of 400 MHz technology. With 533 MHz and 667 MHz memory, typical values for the CL are respectively 4 and 5. Numbers with fractions are possible because data can be clocked at a different rate than commands. With DDR memory, data is clocked at double the speed of commands. For example, 400 MHz DDR memory has a data clock of 400 MHz and a native clock (command and address) of 200 MHz. Thus, CL2.5 memory has a CL of 2.5 command clocks, which is equivalent to five data clocks.
8.2.3 Loaded versus unloaded latency When talking about latency, manufacturers generally refer to CL, which is a theoretical latency and which corresponds to an unloaded latency. A single thread that is executed by a single processor to memory (when data is not in the cache) generates memory access and therefore latency. Because one processor is accessing memory at one time, we talk about unloaded latency.
146
Tuning IBM System x Servers for Performance
As soon as a second processor, or a second thread, concurrently accesses memory, the latency is increased and is defined as loaded latency. In NUMA architecture, if a processor accesses memory locally and another processor accesses the same memory (remotely), there are concurrent memory accesses and, therefore, loaded latency.
8.2.4 STREAM benchmark Many benchmarks exist to test memory performance. Each benchmark acts differently and gives different results because each simulates different workloads. However, one of the most popular and simple benchmarks is STREAM. STREAM is a synthetic benchmark program that measures memory bandwidth in MBps and the computational rate for simple vector kernels. The benchmark is designed to work with larger data sets than the available cache on any given system so that the results are indicative of very large vector style applications. It provides real-world sustained memory bandwidth and not the theoretical peak bandwidth that is provided by vendors. You can find more information about the STREAM benchmark at: http://www.cs.virginia.edu/stream While STREAM gives the memory bandwidth of a given system, a good focus on latency should be made at the same time.
8.3 SMP and NUMA architectures Because different processor manufacturers use a different architecture, memory access to and from processors differs from one solution to another. In the System x world, there are two architectures: SMP and NUMA. We discuss both of these in this section. Both architectures have benefits and limitations, depending on the workload. System x servers with four or fewer Intel processors use the SMP architecture, while System x servers with Opteron processors exclusively use a NUMA architecture.
8.3.1 SMP architecture Systems that are based on the Intel Xeon processor typically use the Symmetric Multiprocessing (SMP) Architecture. The exception in the range of Intel-based System x models is the System x3950 which uses a combination of SMP in each
Chapter 8. Memory subsystem
147
node and NUMA architecture between nodes. SMP systems have a shared front-side bus that connects the processors to a single controller called a north bridge. Each processor communicates with the front-side bus through its bus interface unit (BIU). The CPU accesses memory DIMMs through the separate memory controller or north bridge. The north bridge handles traffic between the processor and memory, controls traffic between the processor and I/O devices, and data traffic between I/O devices and memory. Figure 8-11 shows the central position of the north bridge and the shared front-side bus. These components play the dominant role in determining memory performance. A single processor is capable of saturating the front-side bus, so the second processor could compete for memory bandwidth. The processors also share memory bandwidth with I/O devices, including inter-communication devices.
CPU
CPU
Front-side bus
Memory controller (north bridge)
DIMMs DIMMs
Figure 8-11 An Intel dual-processor memory block
The clock is the internal computer’s components heartbeat. It is the element that defines time. A clock cycle is an upgoing and downgoing electrical signal (alternating high and low voltage). The clock frequency is the number of pulses that are emitted by a clock in one second and is measured in hertz (Hz). Memory accesses are based on clock cycles. Data is sent on rising and falling edges of clock cycles for the DDR and DDR2 (that is what double refers to, in DDR). The processor frequency, the memory bus frequency and the front-side bus frequency are different, however the memory frequency must match the memory bus frequency. The memory frequency can currently go from 200 MHz to 800 MHz. However, it is common to have the front-side bus speed and the memory bus speed matching. The north bridge clock is tied to the speed of the front-side bus, so even as processor clock rates increase, the latency to memory remains virtually the same. The speed of the front-side bus places an upper bound on the rate at
148
Tuning IBM System x Servers for Performance
which a processor can send data to or receive data from memory. In fact, front-side bus bandwidth is often tuned to match the bandwidth of available memory technology at the time. It is expected that a single processor will not saturate the front-side bus over time because the processor has a cache where data most likely to be referenced are stored. Cache reduces the front-side bus pressure, so there is capacity to allow more than one processor to operate on the front-side bus. For example, a two-processor Xeon system can have a front-side bus that is 8 bytes wide and clocked at 800 MHz. Its memory controller has two channels, each 8 bytes wide, to DDR-II 400 memory. This gives the front-side bus and the memory bus 6.4 GBps of bandwidth each (8B * 2 * 400 MHz). Note: The memory frequency drives the memory controller speed (the controller cannot run faster than the memory). As well, the front-side bus limits the processor speed for accessing the memory controller. So, even if the processor speed is increased, the latency and the throughput will be the same, because it is limited by the memory speed.
8.3.2 NUMA architecture Opteron has a different type of architecture—NUMA rather than SMP. The memory controller is integrated into the processor, which can be an advantage for two reasons. First, the memory controller is clocked at the same rate as the processor. So, as the processor speed is increased, the memory controller speed is also increased, which reduces the latency through the memory controller and allows faster access to memory. Second, when a processor is added to a system, more paths to memory are also added. As the demand for memory bandwidth increases due to the additional processor, more bandwidth is available to satisfy that demand. However, with NUMA architecture and with multiple processors, a new concept is raised, remote memory. From a processor point of view, local memory refers to the memory DIMMs that are connected to the processor’s integrated memory controller. Remote memory, as a consequence, refers to memory that is connected to another processors’ integrated memory controller. Note: NUMA architecture is not specific to AMD processors. The IBM X3 servers use a similar technique when clustered. For example, with a two-node System x 3950 server, processors and memory are spread over two nodes. The memory that is located on the first node can be accessed by processors on both the first and second node, as well as all the processors that need to communicate to each other.
Chapter 8. Memory subsystem
149
Figure 8-12 shows the architecture of the Opteron processor. As in Figure 8-11 on page 148, there is a processor core and cache. However, in place of a bus interface unit and an external memory controller, there is an integrated memory controller (MCT), an interface to the processor core (SRQ), three coherent HyperTransportTM (cHT) units and a cross bar switch to handle routing of data, commands, and addresses between them.
AMD Opteron processor Core 0
Core 1
1 MB L2 cache
1 MB L2 cache
System Request
System Request
Integrated Memory controller
DIMMs DIMMs
Crossbar Switch
HT 1
HT 2
HT 3
Opteron processor
Opteron processor
DIMMs DIMMs
HT
Figure 8-12 An AMD dual-processor memory block
Two of the HyperTransport (HT or cHT) units are typically used for connecting to other processors. The third HT unit is to connect to I/O devices. The protocol that is used for routing memory traffic is somewhat more elaborate than what is used for I/O. However, the I/O protocol is a subset, so cHT links can be used for either purpose. Note that every device within the processor package is clocked using a single clock. As the processor clock is increased from one generation or processor speed bin to the next, the memory controller clock is increased automatically at
150
Tuning IBM System x Servers for Performance
the same rate. This increase has the advantage of decreasing the latency of a memory request from the processor core to the memory, which improves access. The disadvantage is that the cHT can be limiting as soon as remote memory access is made.
8.4 The 32-bit 4 GB memory limit A memory address is a unique identifier for a memory location at which a processor or other device can store a piece of data for later retrieval. Each address identifies a single byte of storage. All applications use virtual addresses, not physical. The operating system maps any (virtual) memory requests from applications into physical locations in RAM. When the total amount of virtual memory used by all applications combined exceed the physical RAM installed, the difference is stored in the page file also managed by the operating system. 32-bit CPUs, such as the Intel Xeon, have an architectural limit of only being able to directly address 4 GB of memory. With many enterprise server applications requiring more and more memory, Intel and operating system vendors have developed methods to give applications access to more memory. The first method was implemented by Microsoft with its Windows NT 4.0 Enterprise Edition operating system. Prior to Enterprise Edition, the 4 GB memory space in Windows was divided into 2 GB for the operating system kernel and 2 GB for applications. Enterprise Edition offers the option to allocate 3 GB to applications and 1 GB to the operating system using the 3 GB parameter in the BOOT.INI file. This modification provided a performance improvement of about 20% as measured by the TPC-C benchmark. For more information, see 11.13, “The /3GB BOOT.INI parameter (32-bit x86)” on page 334. The Linux kernel, by default, splits the 4 GB virtual address space of a process in two parts: 3 GB for the user-space virtual addresses and the upper 1 GB for the kernel virtual addresses. The kernel virtual area maps to the first 1 GB of physical RAM and the rest is mapped to the available physical RAM. The potential issue here is that the kernel maps directly all available kernel virtual space addresses to the available physical memory, which means a maximum of 1 GB of physical memory for the kernel. For more information, see the article High Memory in the Linux Kernel, which is available at: http://kerneltrap.org/node/view/2450 For some large enterprise applications, more than 3 GB of memory adds performance benefits. To address more than 4 GB of memory, three addressing schemes were created to access this upper memory: PSE, PAE and, for
Chapter 8. Memory subsystem
151
Windows, AWE. PSE is no longer used. The following sections discuss PAE and AWE.
8.4.1 Physical Address Extension 32-bit operating systems written for the 32-bit Intel processor use a segmented memory addressing scheme. The maximum directly addressable memory is 4 GB (232 ). However, an addressing scheme was created to access memory beyond this limit: the Physical Address Extension (PAE). This addressing scheme is part of the Intel Extended Server Memory Architecture and takes advantage of the fact that the 32-bit memory controller actually has 36 bits that are available for use for memory and L2 addressing. The extra four address bits are normally not used but can be employed along with the PAE or scheme to generate addresses above the 4 GB limit. Using either of these schemes allows access to up to 128 GB of memory. PAE uses a four-stage address generation sequence and accesses memory using 4 KB pages, as shown in Figure 8-13. 32-Bit Linear Address 2
9
9
12
4 Additional Address Lines PTE
4 KB Page
PDE Page Table 512 Entries
PDPE Page Directory Pointer Table
Page Directory 512 Entries
Figure 8-13 PAE-36 address translation
Four reserved bits of control register CR3 pad the existing 32-bit address bus with an additional 4 bits, enabling 36-bit software and hardware addressing to access 64 GB of memory.
152
Tuning IBM System x Servers for Performance
PAE maintains the existing Intel 4 KB memory page definition and requires four levels of redirection to generate each physical memory address. However, as memory capacity increases, using a fixed size 4 KB page results in increased memory management overhead, because the number of memory pages grows as the size of maximum addressable memory increases. Using a larger memory page would reduce the total number of pages and the overhead required to point to any one page, because fewer pages would need to be addressed.
Windows PAE and Address Windowing Extensions PAE is not enabled by default. To use memory beyond 4 GB, you must add the /PAE switch to the corresponding entry in the BOOT.INI file. PAE is supported only on a 32-bit version of the Windows operating system. 64-bit versions of Windows do not support PAE. Note: If you are using a processor with the Data Execution Prevention (DEP) feature (Intel processors refer to this as Execute Disable Bit or XD feature and AMD processors call this the no-execute page-protection processor or NX feature) and have it enabled, then Windows Server 2003 32-bit will automatically enable PAE. To support DEP, Windows will automatically load the PAE kernel no matter how much memory is installed, and you do not have to use the /PAE boot switch in the boot.ini file. The following Windows versions support PAE, with the given amount of physical RAM:
Windows 2000 Advanced Server (8 GB maximum) Windows 2000 Datacenter Server (32 GB maximum) Windows XP (all versions) (4 GB maximum) Windows Server 2003, Standard Edition (4 GB maximum) Windows Server 2003, Enterprise Edition (32 GB maximum) Windows Server 2003, Datacenter Edition (64 GB maximum) Windows Server 2003, Enterprise Edition SP1 (64 GB maximum) Windows Server 2003, Datacenter Edition SP1 (128 GB maximum)
More importantly for developers and application vendors, the virtual memory limits are increased significantly in all versions of Windows Server 2003, x64 Edition.
Chapter 8. Memory subsystem
153
Table 8-3 illustrates the differences between the 32-bit and 64-bit operating systems. Table 8-3 Virtual memory limits
Description
32-bit
64-bit (x64)
Total virtual address space
4 GB
16 TB
Virtual address space per 32-bit application
2 GB (Note 1)
2 GB (Note 2)
Virtual address space per 64-bit process
Not applicable
8 TB
Virtual address space for the OS kernel
2 GB (Note 1)
8 TB
Paged pool
470 MB
128 GB
Non-paged pool
256 MB
128 GB
System cache
1 GB
1 TB
Notes: 1. 3 GB for the application and 1 GB for the kernel if system booted with /3GB switch 2. 4 GB if the 32-bit application has the LARGEADDRESSAWARE flag set (LAA). See “Large
Address Windowing Extensions (AWE) is a set of Windows APIs that implement the PAE functionality of the underlying operating system and allow applications to directly address physical memory above 4 GB. Important: The two BOOT.INI switches /PAE and /3GB do interact with each other and in some circumstances should not be used together. See 11.14.1, “Interaction of the /3GB and /PAE switches” on page 337 for details.
154
Tuning IBM System x Servers for Performance
8.5 64-bit memory addressing The width of a memory address dictates how much memory the processor can address. As shown in Table 8-4, a 32-bit processor can address up to 232 bytes or 4 GB. A 64-bit processor can theoretically address up to 264 bytes or 16 Exabytes (or 16777216 Terabytes). Table 8-4 Relation between address space and number of address bits
Bits (Notation)
Address space
8)
8 (2
256 bytes
16 (216)
65 KB
32 (232)
4 GB
64 (264)
18 Exabytes (EB)
Current implementation limits are related to memory technology and economics. As a result, physical addressing limits for processors are less, as shown in Table 8-5. Table 8-5 Memory supported by processors
Processor
Flat addressing
Addressing with PAE
Intel Xeon MP Gallatin (32-bit)
4 GB (32-bit)
128 GB
Intel EM64T Nocona (64-bit)
64 GB (36-bit)
128 GB in compatibility mode
Intel EM64T Potomac (64-bit)
1 TB (40-bit)
128 GB in compatibility mode
Intel EM64T Cranford (64-bit)
1 TB (40-bit)
128 GB in compatibility mode
Intel EM64T Paxville (64-bit)
1 TB (40-bit)
128 GB in compatibility mode
Intel EM64T Tulsa (64-bit)
1 TB (40-bit)
128 GB in compatibility mode
AMD Opteron (64-bit)
256 TB (48-bit)
128 GB in compatibility mode
Chapter 8. Memory subsystem
155
These values are the limits imposed by the processors. Memory addressing can be limited further by the chipset or supporting hardware in the server. For example, the System x3950 Potomac-based server addresses up to 512 GB of memory in a 32-way configuration when using 4 GB DIMMs—a technology and physical space limitation.
8.6 Advanced ECC memory (Chipkill) All current System x servers implement standard error checking and correcting (ECC) memory. ECC memory detects and corrects any single-bit error. It can also detect double-bit errors but is unable to correct them. Triple-bit and larger errors might not be detected. With the increase in the amount of memory that is used in servers, there is a need for better memory failure protection. As the area of DRAM silicon increases and the density of those DRAM components also increases, there is a corresponding increase in multi-bit failures. This situation means that for larger amounts of memory, there is an increasing propensity for failures that occur to affect more than one data bit at a time and, therefore, overwhelm the traditional single-error correct (SEC) ECC memory module. IBM has developed a new technology colloquially known as Chipkill Protect ECC DIMMs which allow an entire DRAM chip on a DIMM to fail while the system continues to function. These new DIMMs have been designed so that there is no performance degradation over SEC or standard ECC DIMMs. Figure 8-14 shows the results of a failure rate simulation for 32 MB of parity memory, 1 GB of standard SEC ECC, and 1 GB of IBM Advanced ECC memory. The simulation was for three years of continuous operation and showed the significant reduction in failures when using advanced ECC (approximately two orders of magnitude).
156
Tuning IBM System x Servers for Performance
Number of failures in 10,000 systems
1000 900 800 700 600 500 400 300 200 100 0
32 MB Parity [7 failures in 100 systems]
1 GB SEC ECC [9 failures in 100 systems]
1 GB IBM Adv-ECC [6 failures in 10,000 systems]
Results of a BMRS simulation of continuous operation (720 power-on hours) over 36 months using eight 128 MB DIMMs, each with 64 Mb DRAMs in an x4 configuration. Only DRAM failures were considered (not solder, socket failures, for example).
Figure 8-14 Memory failure rate comparison
The capability that the data shows for the memory subsystem is fundamentally the same as RAID technology used for the disk subsystem today. In fact, from a marketing perspective, it could be called RAID-M for Redundant Array of Inexpensive DRAMs for main Memory. This name captures the essence of its function: on-the-fly, automatic data recovery for an entire DRAM failure. IBM is now offering this advanced ECC technology integrated as an option for several members of the System x family. For more information, read the white paper IBM Chipkill Memory, which is available from: http://www.ibm.com/systems/support/supportsite.wss/docdisplay?brandind= 5000008&lndocid=MCGN-46AMQP
8.7 Memory mirroring Memory mirroring is similar to disks RAID array level 1. It means the memory is divided into two equal parts. Actually, the mirroring operates on the ports, which are always designed by pairs. Because the memory pool is divided in two, the total amount of available memory is half the amount of installed memory. Then, if a server is installed with 8 GB RAM and memory mirroring is enabled (in the server’s BIOS), the total available memory seen by the operating system will be 4 GB. It is important to check the DIMMs placement on the board in order to enable this feature (if only one DIMM is installed for example, mirroring will be unavailable).
Chapter 8. Memory subsystem
157
The aim of mirroring the memory is to enhance the server’s availability. If a memory DIMM fails, then the paired DIMM can handle the data. It is a redundant feature that provides failover capabilities but not load-balancing. Tip: Although RAID-1 disks configuration might have a beneficial impact on performance, memory mirroring does not improve memory bandwidth and might even have a negative effect on performance (see Figure 8-19 on page 162). You configure the memory subsystem in the server’s BIOS Setup menu by selecting Advanced Settings → Memory Settings. The window in Figure 8-15 should open. Go to Memory Configuration, and change the default value (Flat) to Mirrored.
Advanced Setup Warning : Memory Settings
DIMM1 and DIMM2 DIMM3 and DIMM4 DIMM5 and DIMM5 DIMM7 and DIMM8 Memory Configuration
[ [ [ [ [
Enabled Enabled Sockets Sockets Flat
] ] are Empty ] are Empty ] ]
PCI Slot/Device Information RSA II Settings Baseboard Management Controller (BMC) Settings
Help <Esc> Exit
<↑> <↓> Move
<→> Next Value <←> Previous Value
Restore Setting Default Setting
Figure 8-15 Enabling memory mirroring
8.8 X3 architecture servers The X3 Architecture servers are the third generation Enterprise X-Architecture servers from IBM. The three servers based on the X3 Architecture are the System x3950, x3850, and x3800. Specific memory options and features are available with the X3 chipset. However, in this section, we focus on the
158
Tuning IBM System x Servers for Performance
performance aspects only.For more details on the X3 Architecture servers, refer to the redbook Planning and Installing the IBM Eserver X3 Architecture Servers, SG24-6797. The X3 Architecture servers implement memory using one to four memory cards, each of which holds four DIMMs, as shown in Figure 8-16. The servers have one or two memory cards installed as standard (model dependant).
The servers supports up to four cards.
Each memory card has four DIMM sockets. DIMMs must be installed in matched pairs.
Models have one or two cards standard. Memory cards can be hot-swapped or hot-added (specific restrictions apply).
D
C A
C
Figure 8-16 Memory card locations (x3950 and x3850 chassis shown)
The memory cards connect directly into the system planar and are powered by two separate memory power buses (two memory cards on each bus). As shown in Figure 8-17, memory cards 1 and 2 are on power bus 1 and memory cards 3 and 4 are on power bus 2. The power arrangement is of particular importance with the hot-swap and hot-add functions.
Chapter 8. Memory subsystem
159
The system memory uses ECC DDR2 DIMMs meeting the PC2-3200 standard. The DIMM layout of the memory cards is also shown in Figure 8-17. Four memory cards (1 or 2 standard)
Hot-swap enabled LED
1 2
Power bus 1 (cards 1 & 2)
Memory port power LED DIMM Socket 1
3 4
DIMM Socket 4
Each card has four DIMM sockets
Power bus 2 (cards 3 & 4)
If memory is installed, card 1 has sockets 1 and 3 filled with 1 GB DIMMs
Figure 8-17 Memory implementation
Table 8-6 shows the standard memory cards and DIMMs. Table 8-6 Standard memory cards and DIMMs
Server
Memory cards
Standard DIMMs
x3800
One (card 1)
Two 1 GB DIMMs or two 512 MB DIMMs (model dependant), installed in memory card 1.
x3850
One (card 1)
Two 1 GB DIMMs, installed in memory card 1.
x3950
One (cards 1 and 2)
Two 1 GB DIMMs, installed in memory card 1. Memory card 2 contains no DIMMs.
x3950 E
Two (cards 1 and 2)
None (all nodes should match).
The configuration rules are as follows: A minimum of one memory card that contains two DIMMs is required for the server to operate. This also applies to the x3950 E. Memory is 2-way interleaved to ensure maximum data throughput. As a result, the DIMMs are always installed in pairs of the same size and type to populate a memory bank. Banks are DIMM sockets 1 and 3 and 2 and 4, as shown in Figure 8-17. The installation sequence for memory cards is 1-2-3-4, for performance-optimized configuration. There are four independent memory ports. Therefore, to optimize performance, you can spread the DIMMs (still
160
Tuning IBM System x Servers for Performance
installed in matched pairs) across all four memory cards before filling each card with two more DIMMs. You can find a more detailed description and the exact sequence for installation in the User’s Guide for each server.
Memory configuration in BIOS Depending on your need, you can configure the system memory in four different ways:
Redundant Bit Steering (RBS), which is the default Full Array Memory Mirroring (FAMM) Hot Add Memory (HAM) High Performance Memory Array (HPMA)
You configure the memory subsystem in the server’s BIOS Setup menu by selecting Advanced Settings → Memory → Memory Array Setting. The window shown in Figure 8-18 opens. Memory Settings Memory Memory Memory Memory Memory
Card 1 Card 2 Card 3 Card 4 Array Setting
[ RBS (Redundant Bit Steering)
]
Figure 8-18 Memory options in BIOS
Table 8-7 shows the choices. Table 8-7 Memory configuration modes in BIOS
Mode
Memory ProteXion
Memorymirroring
Hot-swap memory
Hot-add memory
HPMA (high performance memory array)
Disabled
Disabled
Disabled
Disabled
RBS (redundant bit steering) (default)
Yes
Disabled
Disabled
Disabled
FAMM (full array memory mirroring)
Yes
Yes
Yes
Disabled
HAM (hot-add memory)
Yes
Disabled
Disabled
Yes
Chapter 8. Memory subsystem
161
For best performance, you should select High Performance Memory Array (HPMA). HPMA optimizes the installed memory array on each chassis in the partition for maximum memory performance. Hardware correction (ECC) of a single correctable error per chip select group (CSG) is provided, but RBS is not available. See “Memory ProteXion: Redundant bit steering” on page 163 for a discussion of chip select groups. Figure 8-19 illustrates how memory configuration can have an impact on performance.
4% 10%
32 GB (16x 4 GB DIMMs) mirroring enabled
32 GB (16x 2 GB DIMMs) mirroring disabled
32 GB (16x 2 GB DIMMs) HPMA mode selected
Figure 8-19 Configuration impact on memory operations
However, we recommend that you do not select the HPMA setting in a production environment, because this disables Memory ProteXion.
162
Tuning IBM System x Servers for Performance
Memory ProteXion: Redundant bit steering Redundant bit steering is the technical term for Memory ProteXion. When a single bit in a memory DIMM fails, the function known as redundant bit steering (RBS) moves the affected bit to an unused bit in the memory array automatically, removing the need to perform the ECC correction and thereby returning the memory subsystem to peak performance. The number of RBS actions that can be performed depends on the type of DIMMs installed in the server: A pair of single-rank DIMMs can perform one RBS action. These are the 512 MB and 1 GB DIMMs. A pair of single-ranked DIMMs is also known as a single chip select group (CSG). A pair of double-ranked DIMMs (also known as stacked or double-sided DIMMs) can perform two RBS actions. These are the 2 GB and 4 GB DIMMs. A pair of double-ranked DIMMs is also known as two chip select groups. RBS is supported in both non-mirrored and mirrored configurations. In the X3 Architecture servers, DIMMs are installed in matched pairs in banks. Each memory card installed in the server is comprised of two banks: DIMM sockets 1 and 3 form bank 1 DIMM sockets 2 and 4 form bank 2 Memory errors are handled as follows: If a single-bit error occurs in a CSG, RBS is used to correct the error. If a second single-bit error occurs in the same CSG, the ECC circuitry is used to correct the error. So, for example, if an x3850 is configured with 16x 1 GB DIMMs (which are single-rank), then each pair corresponds to a single CSG, so the server has a total of eight CSGs. This means that the server can survive up to 16 single-bit memory failures—two in each pair of DIMMs (RBS recovery and then ECC). As a second and more complex example, if the same server is installed with eight 1 GB DIMMs (single-rank) and eight 2 GB DIMMs (double-rank), then this means there are a total of 4 + 8 = 12 CSGs (four from the four pairs of 1 GB DIMMs and eight from the eight pairs of 2 GB DIMMs). This means the server can survive up to 24 single-bit memory errors. The first single bit correctable error on a CSG results in an RBS event and RBS log entry from the SMI Handler. The second results in automatic hardware correction and a threshold log entry from the SMI Handler.
Chapter 8. Memory subsystem
163
8.9 IBM Xcelerated Memory Technology The System x3755 implements a new technology IBM Xcelerated Memory Technology that allows all memory DIMMs installed in the server to run at full speed. The x3755 memory subsystem consists of two sets of 4 DIMMs in a daisy chain interconnect. (See Figure 8-20.) term Channel B
term Channel A
4 3 2 1 CPU
Figure 8-20 x3755 memory subsystem
Normally, if a read or write signal is sent down the memory bus to DIMM socket 4, the DIMM furthest away from the CPU, then due to the design of the bus, this signal will be reflected by each DIMM along the bus. This creates additional signals that cause noise and can result in incorrect data reads and writes, which in turn could cause the system to hang. The AMD design specifications says that if you add more than 2 DIMMs on a memory bus then you will have to lower the memory bus speed from 667 MHz to 533 MHz to minimize the effect of the noise. IBM developers, however, found that if you add a circuit to the bus that counteracts the noise, then you can maintain the timing and electrical integrity of the signal. This in turn means that you can keep the bus speed at the higher 667 MHz for all eight DIMMs on each CPU/memory card in the x3755. This IBM unique design allows the x3755 to use the higher memory bus speed of 667 MHz even when using more than two memory DIMMs, thereby improving the memory performance of the x3755 versus the competition.
164
Tuning IBM System x Servers for Performance
8.10 BIOS levels and DIMM placement A important component that affects the memory subsystem performance is the BIOS. The first thing to do is to make sure that the last version of the BIOS is installed. A good example of how a simple BIOS update can improve the memory performance is given by the 1.03 version in the x335. This version introduced a change to enable processor prefetch which increased memory bandwidth (as measured by the Stream Benchmark) by as much as 30%. Pay close attention to the DIMM configuration of your server. Many servers are designed to provide improved performance when an optimal DIMM configuration is employed. However, the optimal DIMM configuration can be different for each machine.
8.11 Memory rules of thumb The rules for memory capacity measurement when upgrading servers that are performing well are straightforward. Usually, the quantity of memory for replacement servers is kept constant or somewhat increased if the number of users and applications does not change. However, this is not always the case. Most of the memory is used typically for file or data cache for the operating system and applications. The operating system requirement of 128 MB to 256 MB can be ignored for memory capacity analysis because this is usually a small fraction of the required server memory. The proper approach is to proportionally scale the amount of memory required for the current number of users based on the expected increase of the number of users. For example, a server with 150 users and 2 GB of memory would need 4 GB to support 300 users. Doubling the number of users requires doubling the amount of server memory. To improve the accuracy of memory requirements, the memory usage of the server that is being replaced must be monitored. There is no guarantee, however, that an existing server always has optimal memory utilization. For Windows environments, you should monitor memory allocation periodically in the Task Manager to determine the amount of total memory that is installed and the average amount of available memory. Total memory minus available memory equals the amount of memory the server is actually using: the working set. Because memory utilization is dynamic, it is best to monitor memory utilization over an extended period of time to arrive at an accurate representation of the memory working set.
Chapter 8. Memory subsystem
165
A useful rule of thumb to determine the amount of memory that is needed to support twice the number of users is to just double the peak working set size and then add 30% as a buffer for growth activity. Servers should be configured so that the average memory utilization does not exceed 70% of installed memory. Generally, 30% is enough extra memory so that the server will not expand storage onto disk or page memory onto disk during periods of peak activity. In any event, when you spot excessive memory utilization and the system starts to page, the best fix is to add memory. The memory rules of thumb are as follows. In general, servers should never regularly page memory to disk (unless the application is performing memory mapped file I/O). Applications that use memory mapped files are Lotus Notes and SAP. For details on memory mapped I/O, see: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dng enlib/html/msdn_manamemo.asp You should rarely have to worry about providing a separate disk for the page device. Paging I/O should occur only occasionally, for example when applications are initializing, but generally not on a continuous basis (unless, of course, you are using an application that employes memory mapped files). In this case, the application might make heavy use of the paging device, and no increase in memory will reduce the amount of paging activity. For these applications (typically Lotus Domino and 32-bit SAP R/3), your only option is to move the page file to a high-speed disk array formatted with a a large stripe size. Most applications will make use of at least 2 GB of virtual memory. Enterprise Edition applications will use up to 3 GB of physical memory. If the application is paging and the maximum amount of memory supported by that application has not been installed, then adding memory is likely to significantly reduce paging. Unfortunately, some applications will continue to page even after the maximum amount of memory is installed. In this case, the only choice is to optimize the paging device by using a high-speed disk array. Average memory utilization should not exceed 70% of available memory. If time is short, simply determine the amount of installed memory on the server being replaced and scale future memory requirements based upon the expected increase in the user community. Memory is relatively inexpensive in comparison to the effort required to accurately predict the exact amount required.
166
Tuning IBM System x Servers for Performance
Performance improvements from adding memory can vary greatly because the improvement depends on so many factors, such as the speed of the disk subsystem, the amount of memory the application requires, the speed of your memory subsystem, the speed of the processor, and so forth. However, Table 8-8 provides a rough idea of the performance improvement you can expect from increasing memory in your server. Table 8-8 Rules of thumb for adding memory
Memory change
Performance gain
1 to 2 GB
20% to 50%
2 to 4 GB
10% to 30%
2 to 6 GB
20% to 40%
4 to 6 GB
10% to 15%
4 to 8 GB
15% to 20%
8 to12 GB
5% to 15%
8 to 16 GB
10% to 20%
16 to 24 GB
5% to 10%
16 to 32 GB
10% to 15%
32 to 64 GB
5% to 10%
Chapter 8. Memory subsystem
167
168
Tuning IBM System x Servers for Performance
9
Chapter 9.
Disk subsystem Ultimately, all data must be retrieved from and stored to disk. Disk accesses are usually measured in milliseconds, while memory and PCI bus operations are measured in nanoseconds or microseconds. Disk operations are typically thousands of times slower than PCI transfers, memory accesses, and LAN transfers. For this reason, the disk subsystem can easily become a major bottleneck for any server configuration. Disk subsystems are also important because the physical orientation of data stored on disk has a dramatic influence on overall server performance. A detailed understanding of disk subsystem operation is critical for effectively solving many server performance bottlenecks. This chapter includes the following sections:
9.1, “Introduction” on page 170 9.2, “Disk array controller operation” on page 172 9.3, “Direct Attached Storage” on page 173 9.4, “Remote storage” on page 182 9.5, “RAID summary” on page 191 9.6, “Factors that affect disk performance” on page 200 9.7, “Disk subsystem rules of thumb” on page 228 9.8, “Tuning with IBM DS4000 Storage Manager” on page 229
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
169
9.1 Introduction A disk subsystem consists of the physical hard disk and the controller. A disk is made up of multiple platters that are coated with a magnetic material to store data. The entire platter assembly mounted on a spindle revolves around the central axis. A head assembly mounted on an arm moves to and fro (linear motion) to read the data that is stored on the magnetic coating of the platter. The linear movement of the head is referred to as the seek. The time it takes to move to the exact track where the data is stored is called seek time. The rotational movement of the platter to the correct sector to present the data under the head is called latency. The ability of the disk to transfer the requested data is called the data transfer rate. In measurement terms, low latency figures are more desirable than high latency figures. With throughput, it is the other way around: the higher the throughput, the better. The most widely used drive technology in servers today is Small Computer System Interface (SCSI) but, as Table 9-1 shows, it is by no means the only standard. Table 9-1 Storage standards
Storage technology
Direct attach or remote storage
Description
SCSI
Direct attach
Probably the most common storage technology currently used in servers. SCSI has just about reached the throughput limits of the current parallel architecture.
SATA (Serial Advanced Technology Attachment)
Direct attach
SATA is set to replace the old Parallel ATA technology and will likely be found in more and more low-end servers over the next few years.
SAS (Serial Attached SCSI)
Direct attach (currently), Remote (future)
SAS is the next evolution of the normal parallel SCSI. As the name suggests, it uses serial instead of parallel technology, resulting in faster bus speeds and longer cable length.
EIDE (Enhanced Integrated Drive Electronics)
Direct attach
EIDE uses Parallel ATA technology. Installed in servers to control peripherals such as CD-ROMs and DVDs.
SSA (Serial Storage Architecture)
Remote
An alternative to Fibre Channel that provides high capacity storage at remote distances to the server.
FC (Fibre Channel)
Remote
Like iSCSI, a method of remote attaching storage to servers, but it does not use the TCP/IP protocol. Has high throughput and low latency.
170
Tuning IBM System x Servers for Performance
Storage technology
Direct attach or remote storage
Description
iSCSI (Internet SCSI)
Remote
SCSI encapsulated in TCP/IP packets to enable connections between servers and remote storage over the existing Ethernet network. Can have high latency on 1 Gb Ethernet networks.
Note that these technologies are divided into two groups, which we discuss in 9.3, “Direct Attached Storage” on page 173 and 9.4, “Remote storage” on page 182. Both EIDE and SCSI use parallel cables to connect the host adapter to the devices. On the face of it, this sounds faster than using serial connections because more data can theoretically be transferred in a single period of time. The problem is that a side effect of increasing the bus speed is the increase in electromagnetic radiation or noise that is emitted from the wires. Because there are physically more wires on parallel cables, there is more noise, and this noise impacts data transmission. This noise means that cable length and bus speeds have been restricted. Developments such as Low Voltage Differential (LVD) SCSI have been introduced to help overcome some of these restrictions, but even these have reached their limits with parallel technology. Recently, there has been a shift away from parallel to serial technology. With serial technology, far fewer wires are needed, typically one pair for transmit and one for receive. Cable pairs are used to enable differential signals to be sent. Serial technology has resulted in smaller and thinner cables with increased bus speeds and longer cable lengths. Note: When describing throughput of a particular storage system, it is common practice to use megabytes per second (MBps) for parallel devices and megabits per second (Mbps) for serial devices. Note that not everyone adheres to this convention. To convert Mbps to MBps, divide by 10 (8 bits per byte plus 2 for a typical overhead). Therefore, 1 Gbps is roughly equal to 100 MBps.
Chapter 9. Disk subsystem
171
9.2 Disk array controller operation The following sequence outlines the fundamental operations that occur when a disk-read operation is performed: 1. The server operating system generates a disk I/O read operation by building an I/O control block command in memory. The I/O control block includes the read command, a disk address called a Logical Block Address (LBA), a block count or length, and the main memory address where the read data from disk is to be placed (destination address). 2. The operating system generates an interrupt to tell the disk array controller that it has an I/O operation to perform. This interrupt initiates execution of the disk device driver. The disk device driver (executing on the server’s CPU) addresses the disk array controller and sends it the address of the I/O control block and a command instructing the disk array controller to fetch the I/O control block from memory. 3. The disk array controller copies the I/O control block from server memory into its local adapter memory. The onboard microprocessor executes instructions to decode the I/O control block command, to allocate buffer space in adapter memory to temporarily store the read data, and to program the SAS controller chip to initiate access to the SAS or SATA disks including the read data. The SAS controller chip is also given the address of the adapter memory buffer that will be used to temporarily store the read data. 4. At this point, the SAS controller sends the read command, along with the length of data to be read, is sent to the target drives that includes the read data over dedicated paths. The SAS controller disconnects from the dedicated path and waits for the next request from the device driver. 5. The target drive begins processing the read command by initiating the disk head to move to the track including the read data (called a seek operation). The average seek time for current high-performance SAS drives is 3 to 5 milliseconds. This time is derived by measuring the average amount of time it takes to position the head randomly from any track to any other track on the drive. The actual seek time for each operation can be significantly longer or shorter than the average. In practice, the seek time depends upon the distance the disk head must move to reach the track including the read data. 6. After the seek time elapses, and the head reaches its destination track, the head begins to read a servo track (adjacent to the data track). A servo track is used to direct the disk head to accurately follow the minute variations of the data signal encoded within the disk surface. The disk head also begins to read the sector address information to identify the rotational position of the disk surface. This step allows the head to know
172
Tuning IBM System x Servers for Performance
when the requested data is about to rotate underneath the head. The time that elapses between the point when the head settles and is able to read the data track, and the point when the read data arrives, is called the rotational latency. Most disk drives have a specified average rotational latency, which is half the time it takes to traverse one complete revolution. It is half the rotational time because on average, the head will have to wait a half revolution to access any block of data on a track. The average rotational latency of a 1000 RPM drive is about 3 milliseconds, while the average rotational latency of a 15 000 RPM drive is about 2 milliseconds. The actual latency depends upon the angular distance to the read data when the seek operation completes, and the head can begin reading the requested data track. 7. When the read data becomes available to the read head, it is transferred from the head into a buffer included on the disk drive. Usually, this buffer is large enough to include a complete track of data. 8. The target drive has the ability to re-establish a dedicated path between itself and the SAS controller. The target drive begins to send the read data into buffers on the adapter SAS controller chip. The adapter SAS controller chip then initiates a direct memory access (DMA) operation to move the read data into a cache buffer in array controller memory. 9. Using the destination address that was supplied in the original I/O control block as the target address, the disk array controller performs a PCI data transfer (memory write operation) of the read data into server main memory. 10.When the entire read transfer to server memory has completed, the disk array controller generates an interrupt to communicate completion status to the disk device driver. This interrupt informs the operating system that the read operation has completed.
9.3 Direct Attached Storage The first four entries in Table 9-1 on page 170 list Direct Attached Storage (DAS) technologies. DAS is connected physically to the server using cables and is available for the server’s exclusive use. Tip: There is no technical reason why Fibre Channel disks could not be attached directly to a server. In practice, Fibre Channel disks are used in enclosures that typically hold many more disks than can be attached directly.
Chapter 9. Disk subsystem
173
9.3.1 SAS The SCSI parallel bus has been the predominant server disk connection technology. However, SAS is beginning to replace it, and System x servers are now offering SAS as the standard storage architecture, both on the board and as options. SAS might be the follow-on to SCSI but, in fact, SCSI is underlying protocol still used. Almost all server disk controllers implement the SCSI communication between the SAS disk controller and disk drives. SCSI is an intelligent interface that allows simultaneous processing of multiple I/O requests. This is the single most important advantage of using SAS controllers on servers. Servers must process multiple independent requests for I/O. The ability of SCSI to concurrently process many different I/O operations makes it the optimal choice for servers. SAS array controllers consist of the following primary components, as shown in Figure 9-1:
PCI bus interface/controller SAS controllers and SAS channels Microprocessor Memory (microprocessor code and cache buffers) Internal bus (connects PCI interface, microprocessor, and SAS controllers)
SAS disk drives
Microprocessor Microcode
SAS controller
Internal Bus
Data buffers Cache
PCI bus controller
Figure 9-1 Architecture of a disk array controller
174
Tuning IBM System x Servers for Performance
SAS protocols and layers SAS uses three protocols to define how transfers are handled between different devices: Serial SCSI Protocol (SSP) supports SAS hard drives and tape devices. SSP is full duplex, so frames can be sent in both directions simultaneously SATA Tunneled Protocol (STP) supports SATA Hard drives. STP is half duplex, so frames can only be sent in one direction at a time Management Protocol (SMP) supports SAS expanders. SMP is a simple protocol that allows initiators to view and configure details about expanders and devices The SAS protocol has the following layers: Application layer The application layer receives commands from the device driver. It then sends the requests (command) to Transport Layer using the appropriate protocol (SSP, STP, or SMP). Transport layer The transport layer is the interface between the application layer and the Port layer. It defines the frame formats. SAS formats are based on Fibre Channel. An example of a common SSP frame format: – – – –
Frame header: 24 bytes Information Unit: 0 to 1024 bytes Fill bytes: 0 to 2 bytes CRC: 4 bytes
Port layer The port layer creates command queues and requests available phys. A phy is the SAS terminology for a port. Ports are abstractions that include phys. Link layer The link layer manages connections between a SAS initiator phy and a SAS target phy. It also arbitrates fairness and deadlocks, and closes connections. Phy layer Out of band (OOB) signaling handles speed negotiation and 8b10b encoding. An OOB signal is a pattern of idle times and burst times. 8b10b coding converts 8 bit bytes into 10-bit data characters for transmission on a wire. Physical layer These are the physical SAS and SATA cables and connectors.
Chapter 9. Disk subsystem
175
SAS and SATA speed negotiation SAS and SATA speed negotiation occurs in the Phy layer. For SAS, to negotiate the connection speed, both devices start at the slowest rate and then increase the rate to find the fastest rate window. The following steps describe the process: 1. Both devices send ALIGN(0)s. 2. If ALIGNED(0)s are received, send ALIGNED(1)s. 3. If ALIGNED(1)s are received, the current rate windows is successful; otherwise, the current rate windows is unsuccessful. 4. If the current rate window is successful, use a faster rate window and repeat steps 1 through 3 until the fastest supported rate is found. 5. Set the connection speed to the fastest supported rate. For SATA, to negotiate the connection speed, both devices start at the fastest rate and then decrease the rate if necessary to find the fastest rate window. The following steps describe the process: 1. SATA target device sends ALIGN primitives at the fastest supported rate. 2. SATA target waits for host to reply with ALIGNs. 3. If no reply is received, the SATA target sends ALIGN primitives at the next slower supported rate. 4. Steps 1 through 3 are repeated until the host replies with ALIGN(0)s, and the fastest supported rate is found. 5. Set the connection speed to the fastest supported rate.
SAS inter-enclosure multi-lane cables SAS inter-enclosure multi-lane cables are used to connect SAS and SAS RAID controllers to external EXP3000 enclosures. Each cable provides four SAS connections. At 3 Gbps per SAS lane, each cable can support up to 12 Gbps of throughput in each direction.
176
Tuning IBM System x Servers for Performance
IBM SAS and SAS RAID controllers use two different types of SAS inter-enclosure multi-lane cables: Mini-SAS Molex (Figure 9-2) connects the PCI Express SAS RAID controller and ServeRAID 8s to an EXP3000 SAS drive enclosure. This cable also connects EXP 3000 enclosures to each other for cascading.
Figure 9-2 Mini-SAS Molex cable
Infiniband-Mini-SAS Molex cable (Figure 9-3) connects MegaRAID 8480E to EXP3000 SAS drive enclosures.
Figure 9-3 Infiniband-Mini-SAS Molex cable
Comparing SAS with SCSI The following lists some of the benefits of SAS over parallel SCSI: Higher performance Unlike parallel SCSI, where devices share a common bus, SAS is a point-to-point architecture where each device connects directly to a SCSI port. By not sharing the same bus, data throughput and reliability are increased.
Chapter 9. Disk subsystem
177
SAS lanes are full duplex. So, each SAS lane supports 3 Gbps of throughput per direction or 6 Gbps of throughput bi-directionally. SAS typically uses four lane cables to connect to external drive enclosures. So, while each parallel SCSI cable can only support a total throughput of 320 MBps, each four lane SAS cable can support a total throughput of 12 Gbps (approximately 1.2 GBps) per direction or 24 Gbps (approximately 2.4 GBps) bi-directionally. Scalability With parallel SCSI, only 14 drives can be attached to each SCSI channel. A four-channel SCSI RAID controller can connect up to 56 drives. With SAS, a single SAS RAID Controller can connect to hundreds of drives when SAS expanders are utilized. Simpler cabling Instead of using large SCSI cables with multiple large connectors, SAS uses thinner cables and smaller connectors. Thinner cables are much easier to manage and improve air circulation to help devices run cooler.
9.3.2 Serial ATA Serial ATA (SATA) is the Serial Advanced Technology Attachment interface specification that offers increased data rate performance over its parallel equivalent, EIDE (now also know as Parallel ATA or PATA) Developed by a group of leading technology vendors, SATA was designed to overcome the performance barriers of Parallel ATA technologies while maintaining their benefits and cost-efficiency. The SATA Working Group introduced the first SATA specification, SATA 1.0, in 2001, with plans for future versions. Tip: The terms Parallel ATA and EIDE are interchangeable and relate to the same technology. For more information about the SATA Working Group, visit: http://www.sata-io.org
Introducing SATA In August 2001, SATA was introduced. Offering a maximum data rate of 150 MBps, SATA 1.0 specifications allowed for thinner, more flexible cables and lower pin counts, thus enabling easier, more flexible cable routing management and the use of smaller connectors than was possible with the existing EIDE/Parallel ATA technology. It is expected that it will completely replace EIDE
178
Tuning IBM System x Servers for Performance
technology in the near future (Ultra™ ATA-100/133 is the last release of EIDE interface technology). In February 2002, a second SATA specification was launched called SATA II. SATA II provided enhancements to the previous specification, with a focus on the area of networked storage. New extensions included solutions for backplane interconnect for hot-swap drive racks, complete enclosure management, and performance enhancements. Note: SATA II was the name of the organization that was formed to create SATA specifications. The group is now called the SATA International Organization (SATA-IO). SATA II does not refer to 3 Gbps data transfer rate for SATA. The increased data transfer rate is one of several features included in SATA specifications subsequent to the SATA 1.0 specification. The SATA road map calls for data rates up to 3 Gbps (300 MBps), improvements to address needs of higher-end networked storage segments, as well as enhancements that address connectivity issues for multiple devices. There are still further plans on the road map for a third generation of SATA that will offer data rates of up to 600 MBps.
Evolutionary improvements SATA addresses several critical design limitations of Parallel ATA such as its maximum throughput of 133 MBps, master and slave configuration limitation, maximum cable length, and its nonexistent road map for growth. Previous improvements to the ATA specification have enabled it to remain competitive with other storage interface technologies; however, it is no longer viable for new applications. The characteristics of SATA make it an ideal solution for new applications like low-cost secondary storage in networked storage environments. New features and benefits introduced with SATA include: Lower voltage SATA operates at 250 millivolts, and Parallel ATA is based on 5 volt signaling. This low voltage results in lower power consumption, which means lower cooling needs, making SATA attractive for multi-drive RAID arrays. Data transfer rates Parallel ATA is limited to data transfer rates of 133 MBps. The initial SATA implementation has a data transfer rate of 150 MBps. Although this transfer rate seems lower, the SATA road map calls for 300 MBps, and then 600 MBps data transfer capability within a few years.
Chapter 9. Disk subsystem
179
Point-to-point connectivity The master and slave shared connectivity approach is replaced with a point-to-point connection scheme supporting only one device per cable. This connectivity allows each drive to communicate directly with the system at any time. Because there is no sharing on the bus, performance scales linearly. Thus, adding a disk on a SATA system gives you the additional maximum throughput of the added disk. Serial transmission Serial transmission is used in many recent technologies including Gigabit Ethernet, USB 2.0, IEEE 1394, and Fibre Channel. In fact, serial is used for most of the fastest data transfer technology and enables SATA to rival SCSI and Fibre Channel in speed. Cyclic redundancy checking (CRC) This provides improved data protection and integrity over PATA and confers to SATA another feature already found in SCSI. Improved performance with hot-swappable drives SATA features greater performance and hot-swappable drives. This feature enables you to swap out a drive without taking the system offline or rebooting. This characteristic makes SATA a viable option for enterprise solutions where system down time is usually not an option. Improved cabling and connector A simplified cabling scheme offers a narrow serial cable with compact connectors for improved connectivity and ventilation, facilitating improved product design and hardware assembly. Practically, the connector size is reduced from 40 pins with Parallel ATA to 7 pins with SATA. Parallel ATA uses 16 separate wires to send 16-bits of data and thus must use a bulky flat cable, which is the cause of electromagnetic interference that compromises data integrity. Backward compatibility with older ATA storage devices SATA is designed to be backward compatible with previous Parallel ATA devices. To system software, SATA is not different from PATA.
SATA and ServeRAID IBM has released its first SATA RAID adapter in the form of the ServeRAID 7t. This adapter has four channels, each capable of connecting to a single drive. The card comes with 64 MB of cache and supports RAID levels 0, 1, and 5. At the time that we wrote this book, this card was supported in only a few entry-level System x servers.
180
Tuning IBM System x Servers for Performance
Figure 9-4 shows the ServeRAID 7t. Notice that the card’s four connectors are along its top edge.
Four SATA ports
Figure 9-4 ServeRAID 7t SATA adapter
For more information about the ServeRAID 7t and other members of the ServeRAID family, see ServeRAID Adapter Quick Reference, TIPS0054, which is available online at: http://www.redbooks.ibm.com/abstracts/tips0054.html
An alternative for enterprise storage The low cost of Parallel ATA has made it the preferred interface technology for personal computers, but it is generally not suited for enterprise applications due to insufficient performance, scalability, and reliability. Enterprise applications have been best served by SCSI and Fibre Channel interface technologies, which provide the necessary performance and reliability for business-critical data and applications. However, one drawback is that customers pay a premium for these types of disk drives and interfaces. Even for less critical data, systems based on Parallel ATA with lower cost drives have unacceptable levels of functionality and reliability. As a result, companies were forced to choose one of these unfavorable options or use older technology (systems previously purchased and rotated out of production environments) for their secondary storage needs. But these previous-generation systems bring high maintenance costs and limited functionality.
Chapter 9. Disk subsystem
181
With the availability of SATA and its established road map for future specification revisions and enhancements, customers have a viable alternative for many enterprise storage applications. As a result, SATA technology is now increasingly found in storage arrays and entry-level servers. SATA is the next generation in the evolution of ATA technology, and satisfies the need for an inexpensive secondary enterprise storage solution while providing some high-end disk system characteristics. This is not to say, however, that SATA is the right answer to every storage requirement. Important: SATA is not the appropriate answer to every storage requirement. For many enterprise applications requiring high performance, and certainly mission-critical and production applications, Fibre Channel disks remain the best choice. For a brief analysis of SATA and Fibre Channel, see “Comparing Fibre Channel with SATA” on page 226. For more information about SATA, see the Introducing IBM TotalStorage FAStT EXP100 with SATA Disks, REDP-3794, which is available from: http://www.redbooks.ibm.com/abstracts/redp3794.html
9.4 Remote storage Remote storage refers to storage that is physically separate from the server and connected by fiber optics, a LAN infrastructure, and in the future through SAS. Remote storage is often shared between multiple servers. Examples of remote storage are shown on the last three rows in Table 9-1 on page 170. In this section, we cover two of these (SSA is not covered): 9.4.2, “Fibre Channel” on page 186 9.4.3, “iSCSI” on page 189 One point to remember is that Fibre Channel and iSCSI are used to transfer the data between the server and remote storage device. The remote storage device might actually be using Fibre Channel, SAS, or SATA disks.
9.4.1 Differences between SAN and NAS Before describing the different remote disk technologies, it is worth discussing Storage Attached Network (SAN) and Network Attached Storage (NAS) and how they differ. Both of our remote storage topics, Fibre Channel and iSCSI, are forms of SANs. Although iSCSI works over the existing Ethernet network, it is not an NAS system.
182
Tuning IBM System x Servers for Performance
Tip: If you want to read more about SAN and NAS implementations, beyond what is said below, review the redbook IBM System Storage Solutions Handbook, SG24-5250, which you can find online at: http://www.redbooks.ibm.com/abstracts/sg245250.html
SAN A SAN is a specialized, dedicated high-speed storage network. Servers, switches, hubs, and storage devices can attach to the SAN. It is sometimes called the network behind the servers. Like a LAN, a SAN allows any-to-any connection across the network, using interconnect elements such as routers, gateways, hubs, and switches. Fibre Channel is the de facto SAN networking architecture, although you can use other network standards. Fibre Channel is a multi-layered network, based on a series of ANSI standards. These define characteristics and functions for moving data across the network. As with other networks, information is sent in structured packets or frames, and data is serialized before transmission. However, unlike other networks, the Fibre Channel architecture includes a significant amount of hardware processing. The maximum data rate currently supported is 4 Gbps or 400 MBps full duplex. However, a SAN implementation does not come without a price. Because of the complexities involved, a SAN can be an expensive investment. Storage management becomes a consideration. A high level of skill is needed to maintain and manage a SAN. It is therefore worth investing a significant amount of time in planning the implementation of a SAN.
Designing a SAN Designing and implementing a SAN requires a knowledge of the fundamental storage principles that are needed to create a storage subsystem that can handle the I/O requirements of an enterprise production environment. For a review of storage fundamentals see 9.6, “Factors that affect disk performance” on page 200. You should also consider how multiple sets of users, applications, and servers accessing the storage pool will affect performance. Conceptually split the SAN into three different zones: Backend zone The backend zone is defined as the hardware from the hard disk drives to the backend of the remote storage controllers. The backend zone must include the optimal drive technology, number of drives, and sufficient bandwidth capacity up to the remote storage to satisfy the I/O requests from all servers and applications that have access to a particular backend zone.
Chapter 9. Disk subsystem
183
The potential for creating a bottleneck in a SAN is very high in the backend zone. One reason for this is the backend technology such as hard disk drives and disk drive enclosures are typically the last components to complete a technology jump such as 2 Gbit to 4 Gbit Fibre Channel. Furthermore, it is very expensive and time consuming to upgrade backend technology compared to frontend and middle zone technology. Therefore, the front and middle zones of a SAN might include 4 Gbit FC technology while the backend remains at 2 Gbit FC technology. Because SANs must accommodate the full spectrum of different workload characteristics, the potential for a bottleneck in streaming environments such as backups, restores, and table scans is high. At a minimum, a user might not realize the full potential of the SANs performance capabilities if one zone is populated with inferior technology. Important: The hardware in the backend zone is critical to ensuring optimal performance of a SAN, because the backend zone is where data begins its journey up to the servers. If the drive technology is insufficient or if the bandwidth technology is inferior to the bandwidth technology in the middle and frontend zones, the performance of the entire SAN could potentially be gated. Middle zone The middle zone includes the hardware from the remote storage controllers up to the backend of any switches, hubs, or gateway hardware. Sufficient bandwidth capacity must exist in the middle zone to allow for sustainable throughput coming from the backend zone en route to the frontend zone. Frontend zone The frontend zone includes the hardware from the front of any switches, hubs, or gateways up to and including the host bus adapter (HBA), host channel adapter (HCA), or adapter used to feed the servers replies from I/O requests. Again, sufficient bandwidth capacity must exist in the frontend zone to allow for sustainable throughput coming from the middle zone. For example, if four 4 Gbit connections exist in the middle zone, and there are only two 4 Gbit host connections in the frontend zone, a bottleneck could easily develop in a streaming intensive workload environment. In addition to considering fundamental storage principles and the hardware in the three different zones, it is just as important to consider the load placed on the SAN. The aggregate load on the SAN must be balanced across hard drives, remote storage controllers, links, and switches up through each zone of the SAN. Unbalanced loads will cause portions of the SAN to be under-utilized, and other
184
Tuning IBM System x Servers for Performance
portions of the SAN to be over-utilized. See 9.8, “Tuning with IBM DS4000 Storage Manager” on page 229 for information about a tool to monitor load balance on a FC SAN.
NAS Storage devices which optimize the concept of file sharing across the network have come to be known as NAS. NAS solutions use the mature TCP/IP network technology of the Ethernet LAN. Data is sent to and from NAS devices over the LAN using TCP/IP protocol. By making storage devices LAN addressable, the storage is freed from its direct attachment to a specific server, and any-to-any connectivity is facilitated using the LAN fabric. In principle, any user running any operating system can access files on the remote storage device. This is done by means of a common network access protocol, for example, NFS for UNIX servers, and CIFS for Windows servers. In addition, a task, such as backup to tape, can be performed across the LAN, using software like Tivoli® Storage Manager, enabling sharing of expensive hardware resources, such as automated tape libraries, between multiple servers. A storage device cannot just attach to a LAN. It needs intelligence to manage the transfer and the organization of data on the device. The intelligence is provided by a dedicated server to which the common storage is attached. It is important to understand this concept. NAS comprises a server, an operating system, plus storage which is shared across the network by many other servers and clients. So an NAS is a device, rather than a network infrastructure, and shared storage is attached to the NAS server. One of the key differences between an NAS disk device and Direct Attached Storage or other network storage solutions, such as SAN or iSCSI, is that all I/O operations use file-level I/O protocols. File I/O is a high-level type of request which, in essence, specifies only the file to be accessed but does not directly address the storage device. This directly address the storage device is done later by other operating system functions in the remote NAS appliance. A file I/O specifies the file and also indicates an offset into the file. For instance, the I/O might specify “Go to byte ‘1000’ in the file (as though the file were a set of contiguous bytes), and read the next 256 bytes beginning at that position.” Unlike block I/O, a file I/O request has no awareness of disk volume or disk sectors. Inside the NAS appliance, the operating system keeps tracks of where files are located on disk. The OS issues a block I/O request to the disks to fulfill the file I/O read and write requests it receives. In summary, the network access methods, NFS and CIFS, can only handle file I/O requests to the remote file system, which is located in the OS of the NAS device. I/O requests are packaged by the initiator into TCP/IP protocols to move
Chapter 9. Disk subsystem
185
across the IP network. The remote NAS file system converts the request to block I/O and reads or writes the data to the NAS disk storage. To return data to the requesting client application, the NAS appliance software re-packages the data in TCP/IP protocols to move it back across the network. A database application which is accessing a remote file located on an NAS device, by default, is configured to run with file system I/O. It cannot use a raw I/O to achieve improved performance. Because NAS devices attach to mature, standard LAN infrastructures, and have standard LAN addresses, they are, typically, extremely easy to install, operate and administer. This plug and play operation results in low risk, ease of use, and fewer operator errors, so it contributes to a lower cost of ownership.
9.4.2 Fibre Channel Fibre Channel introduces different techniques to attach storage to servers and as a result, it has unique performance issues that affect the overall performance of a server. This section provides a brief introduction to the motivation behind Fibre Channel, explains how Fibre Channel affects server performance, and identifies important issues for configuring Fibre Channel for optimal performance. SCSI has been the standard for server disk attachment for the last 10 years. However, SCSI technology has recently been under stress as it attempts to satisfy the I/O demands of current high-performance 4- and 8-way servers. Some of the fundamental issues with SCSI are its parallel cable design, which limits cable length, transfer speed, and the maximum number of drives that can be attached to the cable. Another significant limitation is that a maximum of two systems can share devices that are attached to one SCSI bus, which is a significant limitation when using SCSI for server clustering configurations. Fibre Channel was designed to be a transport for both network traffic and an I/O channel for attaching storage. In fact, the Fibre Channel specification provides for many protocols such as 802.2, IP (Internet Protocol) and SCSI. Our discussion in this book is limited to its use for disk storage attachment. Fibre Channel provides low latency and high throughput capabilities. As a result, Fibre Channel is rapidly becoming the next-generation I/O technology used to connect servers and high-speed storage. Fibre Channel addresses many of the shortcomings of SCSI with improvement in the following areas:
186
Cable distance Bandwidth Reliability Scalability
Tuning IBM System x Servers for Performance
The parallel cable used for Ultra320 SCSI limits cable distances to 25 meters or shorter because of electromagnetic effects that impact signal integrity as cable length increases. Parallel cables such as the type that are used by SCSI tend to have signal interference problems because of electromagnetic coupling that occurs between parallel signals traversing the wires. Serial technologies use fewer signals, typically two or four, compared to as many as 68 for SCSI. Fewer signal lines means less electromagnetic energy is emitted and less total signal interference from coupling of the electromagnetic energy into adjacent wires. Lower signal interference allows the serial cable to transfer data at much higher rates than is possible using a parallel connection. Fibre Channel provides the capability to use either a serial copper or fiber optic link to connect the server with storage devices. Fiber optic technology allows for storage to be located a maximum distance of up to 10 kilometers away from the attaching server. A significant advantage of Fibre Channel is its ability to connect redundant paths between storage and one or more servers. Redundant Fibre Channel paths improve server availability because cable or connector failures do not cause server down time because storage can be accessed by a redundant path. In addition, both Fibre Channel and SCSI throughput can scale by utilizing multiple channels or buses between the servers and storage. In addition to a simpler cable scheme, Fibre Channel offers improved scalability due to several very flexible connection topologies. Basic point-to-point connections can be made between a server and storage devices providing a low-cost simple stand-alone connection. Fibre Channel can also be used in both loop and switch topologies. These topologies increase server-to-storage connection flexibility. The Fibre Channel loop allows up to 127 devices to be configured to share the same Fibre Channel connection. A device can be a server, storage subsystem, drive enclosure, or disk. Fibre Channel switch topologies provide the most flexible configuration scheme by theoretically providing the connection of up to 16 million devices. The Fibre Channel specification provides many possibilities for how Fibre Channel is configured, but we will confine our discussion to the implementation of the IBM TotalStorage DS4000 series RAID controllers. The TotalStorage DS4000 controller operation can be conceptualized by combining LAN and disk array controller operations. Figure 9-5 illustrates the primary components in the DS4000 configuration. The important factors that contribute to performance are brought about because the RAID controller and storage are attached to the server by a Fibre Channel link.
Chapter 9. Disk subsystem
187
EX P-15 System x server Optional second FC adapter Fibre Channel Fibre Fibre Channel Channel host adapters host adapters H ost Adapter
Optional second RAID controller
FC-AL
RAIDFibre DS4000 Channel RAID Controller controller
FC bandwidth
FC-to-disk bandwidth
Figure 9-5 IBM TotalStorage DS4000 connectivity
This configuration introduces two factors which contribute to overall Fibre Channel performance: The throughput of the Fibre Channel links, shown as the FC bandwidth arrow. These links are limited to a certain data rate. Current Fibre Channel optical links can sustain either 1 Gbps, 2 Gbps, and 4 Gbps. Therefore, it is imperative that a Fibre Channel storage subsystem include enough Fibre Channel links to sustain a specific bandwidth requirement. For example, if a streaming media environment requires a data transfer rate of 420 MBps from the disks to the host, than a minimum of three 2 Gbps Fibre Channel links or two 4 Gbit FC links are required between the RAID controller and the host. Anything less does not meet the 420 MBps specification. Simply providing the link bandwidth that is required to sustain a given data transfer rate will not guarantee that the storage subsystem is capable of achieving the required data transfer rate. The aggregate throughput of the RAID controller and link combination, shown as the FC-to-disk bandwidth arrow. This throughput determines whether the storage subsystem can meet a given data transfer rate. This value includes the link data rate from the drives to the RAID controller and the performance capability of the RAID controller. Consider the 420 MBps streaming media requirement that is used to explain the first factor. If the links from the drives to the RAID controller were only 1 Gbps, then the maximum theoretical aggregate throughput of the RAID
188
Tuning IBM System x Servers for Performance
controller and the links between the drives and the RAID controller is limited to 400 MBps no matter what the RAID controller is capable of achieving. This analysis is particularly critical for data streaming environments.
9.4.3 iSCSI iSCSI is an industry standard that allows SCSI block I/O protocols (commands, sequences and attributes) to be sent over a network using the TCP/IP protocol. This is analogous to the way SCSI commands are already mapped to Fibre Channel, parallel SCSI, and SSA media (do not confuse this with the SCSI cabling transport mechanism; here we are addressing protocols). iSCSI is a network transport protocol for SCSI that operates on top of TCP. It encapsulates SCSI protocols into a TCP/IP frame, so that storage controllers can be attached to IP networks. Unlike Fibre Channel, iSCSI uses the existing Gigabit Ethernet LAN as a medium to transfer data from the iSCSI appliance, known as the target, to the file or application server. At the server end, either a software iSCSI driver or a dedicated iSCSI adapter can be used to encapsulate the iSCSI blocks. This is know as the initiator. If a software initiator is used, then the iSCSI traffic will be transmitted through the existing network adapters. If a hardware initiator is installed then it will need its own Ethernet network connection.
Performance So what sort of performance can we expect from an iSCSI SAN? Figure 9-6 shows the comparative performance between directly attached SCSI, Fibre Channel, and iSCSI.
64 K Sequential Read Throughput 300
Throughput (MBps)
250 200 150 100 50 0
SCSI
Fibre Channel
iSCSI
Figure 9-6 64 K Sequential Read Throughput comparison
Chapter 9. Disk subsystem
189
As the figure shows, directly attached SCSI is the fastest with Fibre Channel not far behind. The iSCSI throughput is about 50%, which is the speed of the Fibre Channel system. Why is this? To some extent, iSCSI is limited by the speed of 1 Gb Ethernet. However, Figure 9-7 shows that the latency value for iSCSI is approximately twice that of Fibre Channel. This high latency, which is a result of the TCP/IP protocol, is the main reason for the poor throughput of iSCSI.
64K Sequential Read Latency 5
Latency (ms)
4
3
2
1
0
SCSI
Fibre Channel
iSCSI
Figure 9-7 64K Sequential Read Latency comparison
iSCSI is a new technology. The performance figures will no doubt improve as it develops. One of the interesting features of iSCSI is its ability to boot. This feature will initially only be available from the hardware initiator, but it does offer the possibility of diskless servers.
Security Many IT managers would have serious reservations about running mission-critical corporate data on an IP network which is also handling other traffic. iSCSI introduces the possibility of an IP network SAN, which could be shared. To alleviate these worries, iSCSI can be encrypted using IPSEC. For information about iSCSI from a networking point of view, see 10.4, “Internet SCSI (iSCSI)” on page 283.
190
Tuning IBM System x Servers for Performance
9.5 RAID summary Most of us have heard of RAID (Redundant Array of Independent Disks) technology. Unfortunately, there is still significant confusion about the performance implications of each RAID strategy. This section presents a brief overview of RAID and the performance issues as they relate to commercial server environments. RAID is a collection of techniques that treat multiple, inexpensive disk drives as a unit, with the object of improving performance and reliability. Table 9-2 lists the RAID levels offered by RAID controllers in IBM System x servers. Table 9-2 RAID summary
RAID level
Fault tolerant?
Description
RAID-0
No
All data evenly distributed (striping) to all drives. See 9.5.1, “RAID-0” on page 192.
RAID-1
Yes
A mirrored copy of one drive to another drive (two disks). See 9.5.2, “RAID-1” on page 192.
RAID-1E
Yes
All data is mirrored (more than two disks). See 9.5.3, “RAID-1E” on page 193.
RAID-4
Yes
One disk in array is shared for parity, and all other drives in array are striped with data. See 9.5.4, “RAID-4” on page 194.
RAID-5
Yes
Distributed checksum. Both data and parity are striped across all drives. See 9.5.5, “RAID-5” on page 194.
RAID-5E
Yes
Distributed checksum and hot-spare. Data, parity and hot-spare are striped across all drives. See 9.5.6, “RAID-5EE and RAID-5E” on page 195.
RAID-5EE
Yes
Distributed checksum and hot-spare. Data, parity and hot-spare are striped across all drives. Same as RAID-5E except has faster rebuild times. See 9.5.6, “RAID-5EE and RAID-5E” on page 195.
RAID-6
Yes
Distributed checksum. Both data and parity are striped across all drives - twice, to provide two-drive failure fault tolerance. See 9.5.7, “RAID-6” on page 198.
RAID-10
Yes
Striping (RAID-0) across multiple RAID-1 arrays. See 9.5.8, “Composite RAID levels” on page 199.
RAID-50
Yes
Striping (RAID-0) across multiple RAID-5 arrays. See 9.5.8, “Composite RAID levels” on page 199
Chapter 9. Disk subsystem
191
9.5.1 RAID-0 RAID-0 is a technique that stripes data evenly across all disk drives in the array. Strictly, it is not a RAID level, because no redundancy is provided. On average, accesses is random, thus keeping each drive equally busy. SCSI has the ability to process multiple, simultaneous I/O requests, and I/O performance is improved because all drives can contribute to system I/O throughput. Because RAID-0 has no fault tolerance, when a single drive fails, the entire array becomes unavailable. RAID-0 offers the fastest performance of any RAID strategy for random commercial workloads. RAID-0 also has the lowest cost of implementation because redundant drives are not supported. Logical view 1 2 3 4 5 6 7 8 9
1
2
3
4
5
6
7
8
9
RAID-0 - Physical view
Figure 9-8 RAID-0: All data evenly distributed across all drives but no fault tolerance
9.5.2 RAID-1 RAID-1 provides fault tolerance by mirroring one drive to another drive. The mirror drive ensures access to data should a drive fail. RAID-1 also has good I/O throughput performance compared to single-drive configurations because read operations can be performed on any data record on any drive contained within the array. Most array controllers (including the ServeRAID family) do not attempt to optimize read latency by issuing the same read request to both drives in the mirrored pair. The drive in the pair that is least busy is issued the read command, leaving the other drive to perform another read operation. This technique ensures maximum read throughput.
192
Tuning IBM System x Servers for Performance
Write performance is somewhat reduced because both drives in the mirrored pair must complete the write operation. For example, two physical write operations must occur for each write command generated by the operating system. RAID-1 offers significantly better I/O throughput performance than RAID-5. However, RAID-1 is somewhat slower than RAID-0.
1
1'
2
2'
3
3'
RAID-1 - Physical view Figure 9-9 RAID-1: Fault-tolerant; a mirrored copy of one drive to another drive
9.5.3 RAID-1E RAID-1 Enhanced, or more simply, RAID-1E, is only implemented by the IBM ServeRAID adapter and allows a RAID-1 array to consist of three or more disk drives. “Regular” RAID-1 consists of exactly two drives. The data stripe is spread across all disks in the array to maximize the number of spindles that are involved in an I/O request to achieve maximum performance. RAID-1E is also called mirrored stripe, as a complete stripe of data is mirrored to another stripe within the set of disks. Like RAID-1, only half of the total disk space is usable; the other half is used by the mirror.
1
2
3
3'
1'
2'
4
5
6
RAID-1E - Physical view Figure 9-10 RAID-1E: Mirrored copies of each drive
Because you can have more than two drives (up to 16), RAID-1E will outperform RAID-1. The only situation where RAID-1 performs better than RAID-1E is in the reading of sequential data. The reason for this is that when a RAID-1E reads sequential data off a drive, the data is striped across multiple drives. RAID-1E interleaves data on different drives, so seek operations occur more frequently during sequential I/O. In RAID-1, data is not interleaved, so fewer seek operations occur for sequential I/O.
Chapter 9. Disk subsystem
193
RAID-10 can also be used to increase the number of drives in a RAID-1 array. This technique consists of creating a RAID-0 array and mirroring it with another collection of similar-capacity drives. Thus, you can configure two sets of five drives each in a RAID-0 configuration, and mirror the two sets of drives. This configuration would deliver the same performance for most commercial applications as a 10-drive RAID-1E configuration, but RAID-1E lacks one added benefit. Each of the RAID-0 arrays in the RAID-10 configuration can be contained in two different drive enclosures. Thus, if one drive enclosure fails because of a bad cable or power supply, the other mirror set can provide data access. With RAID-10, an entire set of drives (five in this case) can fail and the server can still access the data.
9.5.4 RAID-4 RAID-4 provides single-drive fault tolerance by striping data blocks across all drives in an array except for one, which is shared for parity. Figure 9-11 illustrates the physical layout of data and party on an array consisting of four drives.
1
2
3
1-3 parity
4
5
6
4-6 parity
7
8
9
7-9 parity
RAID-4 - Physical view
Figure 9-11 RAID-4: data striped across all drives except one, which is shared for parity
If a data disk fails, the missing data is calculated on the fly using an exclusive-or operation, thus ensuring continued access to data with a single disk failure. All of the data on the failed drive is recalculated after the failed disk is replaced. Failure of the parity disk does not result in a loss of any file system data.
9.5.5 RAID-5 RAID-5 offers an optimal balance between price and performance for most commercial server workloads. RAID-5 provides single-drive fault tolerance by implementing a technique called single equation single unknown. This technique implies that if any single term in an equation is unknown, the equation can be solved to exactly one solution. The RAID-5 controller calculates a checksum (parity stripe in Figure 9-12) using a logic function known as an exclusive-or (XOR) operation. The checksum is the XOR of all data elements in a row. The XOR result can be performed quickly by the RAID controller hardware and is used to solve for the unknown data element.
194
Tuning IBM System x Servers for Performance
In Figure 9-12, addition is used instead of XOR to illustrate the technique: stripe 1 + stripe 2 + stripe 3 = parity stripe 1-3. Should drive one fail, stripe 1 becomes unknown and the equation becomes X + stripe 2 + stripe 3 = parity stripe 1-3. The controller solves for X and returns stripe 1 as the result. A significant benefit of RAID-5 is the low cost of implementation, especially for configurations requiring a large number of disk drives. To achieve fault tolerance, only one additional disk is required. The checksum information is evenly distributed over all drives, and checksum update operations are evenly balanced within the array.
1
2
3
1-3 parity
4
5
4-6 parity
6
7
7-9 parity
8
9
RAID-5 - Physical view
Figure 9-12 RAID-5: both data and parity are striped across all drives
However, RAID-5 yields lower I/O throughput than RAID-0 and RAID-1. This is due to the additional checksum calculation and write operations required. In general, I/O throughput with RAID-5 is 30% to 50% lower than with RAID-1 (the actual result depends upon the percentage of write operations). A workload with a greater percentage of write requests generally has a lower RAID-5 throughput. RAID-5 provides I/O throughput performance similar to RAID-0 when the workload does not require write operations (read only).
9.5.6 RAID-5EE and RAID-5E IBM research invented RAID-5E. It is a technique that distributes the hot-spare drive space over the n+1 drives that comprise the RAID-5 array plus standard hot-spare drive. RAID-5E was first implemented in ServeRAID firmware V3.5 and was introduced to overcome the long rebuild times that are associated with RAID-5E in the event of a hard drive failure. Some older ServeRAID adapters only support RAID-5E. For more information, see ServeRAID Adapter Quick Reference, TIPS0054, which is available online at: http://www.redbooks.ibm.com/abstracts/tips0054.html Adding a hot-spare drive to a server protects data by reducing the time spent in the critical state. This technique does not make maximum use of the hot-spare drive because it sits idle until a failure occurs. Often many years can elapse before the hot-spare drive is ever used. IBM invented a method to use the hot-spare drive to increase performance of the RAID-5 array during typical
Chapter 9. Disk subsystem
195
processing and preserve the hot-spare recovery technique. This method of incorporating the hot spare into the RAID array is called RAID-5EE.
2
1
3
4
1-4 parity
5
6
7
5-8 parity
8
Hot spare
Hot spare
Hot spare
Hot spare
Hot spare
RAID-5E - Physical view
Figure 9-13 RAID-5E: The hot spare is integrated into all disks, not a separate disk
RAID-5EE is designed to increase the normal operating performance of a RAID-5 array in two ways: The hot-spare drive includes data that can be accessed during normal operation. The RAID-5 array now has an extra drive to contribute to the throughput of read and write operations. Standard 10 000 RPM drives can perform more than 100 I/O operations per second so the RAID-5 array throughput is increased with this extra I/O capability. The data in RAID-5EE is distributed over n+1 drives instead of n as is done for RAID-5. As a result, the data occupies fewer tracks on each drive. This has the effect of physically utilizing less space on each drive, keeping the head movement more localized and reducing seek times. Together, these improvements yield a typical system-level performance gain of about 10% to 20%. One disadvantage of RAID-5EE is that the hot-spare drive cannot be shared across multiple physical arrays as can be done with standard RAID-5 plus a hot-spare. This RAID-5 technique is more cost-efficient for multiple arrays because it allows a single hot-spare drive to provide coverage for multiple physical arrays. It reduces the cost of using a hot-spare drive, but the downside is the inability to handle separate drive failures within different arrays. IBM ServeRAID adapters offer increased flexibility by offering the choice to use either standard RAID-5 with a hot-spare or the newer integrated hot-spare provided with RAID-5EE.
196
Tuning IBM System x Servers for Performance
While RAID-5EE provides a performance improvement for most operating environments, there is a special case where its performance can be slower than RAID-5. Consider a three-drive RAID-5 with hot-spare configuration, as shown in Figure 9-14. This configuration employs a total of four drives, but the hot-spare drive is idle. Thus, for a performance comparison, it can be ignored. A four-drive RAID-5EE configuration would have data and checksum on four separate drives. ServeRAID adapter Adapter cache 16 KB write operation
Step 2: calculate checksum
Step 1 Step 3: write data
Step 4: write checksum
16 KB write operation
RAID-5 with hot-spare 8 KB stripe
8 KB
8 KB
8 KB
Figure 9-14 Writing a 16 KB block to a RAID-5 array with an 8 KB stripe size
Referring to Figure 9-14, whenever a write operation is issued to the controller that is two times the stripe size (for example, a 16 KB I/O request to an array with an 8 KB stripe size), a three-drive RAID-5 configuration would not require any reads because the write operation would include all the data needed for each of the two drives. The checksum would be generated by the array controller (step 2) and immediately written to the corresponding drive (step 4) without the need to read any existing data or checksum. This entire series of events would require two writes for data to each of the drives storing the data stripe (step 3) and one write to the drive storing the checksum (step 4), for a total of three write operations. Contrast these events to the operation of a comparable RAID-5EE array which includes four drives, as shown in Figure 9-15. In this case, in order to calculate the checksum, a read must be performed of the data stripe on the extra drive (step 2). This extra read was not performed with the three-drive RAID-5 configuration, and it slows the RAID-5EE array for write operations that are twice the stripe size.
Chapter 9. Disk subsystem
197
ServeRAID adapter Adapter cache 16 KB write operation
Step 3: calculate checksum
Step 1 Extra step
Step 4: write data
Step 2: read data
Step 5: write checksum
16 KB write operation
RAID-5EE with integrated hot-spare 8 KB stripe
8 KB
8 KB
8 KB
8 KB
Figure 9-15 Writing a 16 KB block to a RAID-5EE array with a 8 KB stripe size
You can avoid this issues with RAID-5E by selecting the proper stripe size. By monitoring the average I/O size in bytes or by knowing the I/O size that is generated by the application, you can select a large enough stripe size so that this performance degradation rarely occurs.
9.5.7 RAID-6 RAID-6 provides fault tolerance that allows for two drives to fail, or a single drive failure and subsequent bad block failure. The fault tolerance of the second drive failure is achieved by implementing a second distributed parity method across all of the drives in a RAID-6 array. RAID-6 requires a minimum of four drives. The two-drive fault tolerance provided by RAID-6 is computed using Galois field algebra. If you are interested, refer to books on group and ring theory for an in-depth examination of Galois field algebra. The rebuild process for a single drive failure is not as complex as the rebuild process for two-drive failures. Remember that performance degrades during rebuild time due to the RAID controller devoting cycles to restoring data as well as simultaneously processing incoming I/O requests. Therefore, the user must decide if the extra fault tolerance is worth degraded performance for a longer rebuild time following the failure of two drives. Alternatively, with the increasing popularity of less expensive, less robust hard disk drive technology such as SATA, a RAID-6 configuration might be worth the longer rebuild times. In addition, the ever increasing capacities of hard disk
198
Tuning IBM System x Servers for Performance
drives could potentially increase the chances of another disk failure or bad block failure during longer rebuild times due to the larger capacity. As always, performance must be weighed against potential downtime due to drive failures.
9.5.8 Composite RAID levels The ServeRAID adapter family supports composite RAID levels. This means that it supports RAID arrays that are joined together to form larger RAID arrays. The ServeRAID firmware only supports 8 arrays, therefore only 16 drives (8 spanned RAID arrays) are supported in RAID-1. RAID-00, RAID-1E0, and RAID-50 can support more total drives as long as the total number of arrays is 8 or less. For example, RAID-10 is the result of forming a RAID-0 array from two or more RAID-1 arrays. With two SCSI channels each supporting 15 drives, the ServeRAID 6M can theoretically have up to 30 drives in one array. With the EXP300 and the EXP400, the limit is 56 disks. Figure 9-16 illustrates a ServeRAID RAID-10 array.
1
1'
1
1'
1
1'
2
2'
2
2'
2
2'
3
3'
3
3'
3
3'
1
2
3
4
5
6
7
8
9
RAID-10 - Physical view (striped RAID-1) Figure 9-16 RAID-10: a striped set of RAID-1 arrays
Chapter 9. Disk subsystem
199
Likewise, Figure 9-17 shows a striped set of RAID-5 arrays.
1
2
3
1-3 parity
1
2
3
1-3 parity
4
5 7-9 parity
4-6 parity
6
4
4-6 parity
6
8
9
7
5 7-9 parity
8
9
7
1
2
3
4
5
6
RAID-50 - Physical view (striped RAID-5)
Figure 9-17 RAID-50: a striped set of RAID-5 arrays
Many of the ServeRAID adapters supports the combinations that are listed in Table 9-3. Table 9-3 Composite RAID levels supported by ServeRAID-4 adapters
RAID level
The sub-logical array is
The spanned array is
RAID-00
RAID-0
RAID-0
RAID-10
RAID-1
RAID-0
RAID-1E0
RAID-1E
RAID-0
RAID-50
RAID-5
RAID-0
9.6 Factors that affect disk performance Many factors can affect disk performance. The most important considerations (in order of importance) for configuring ServeRAID and Fibre Channel solutions are:
200
9.6.1, “RAID strategy” on page 201 9.6.2, “Number of drives” on page 202 9.6.3, “Active data set size” on page 203 9.6.4, “Drive performance” on page 205 9.6.5, “Logical drive configuration” on page 206 9.6.6, “Stripe size” on page 207 9.6.7, “SCSI bus organization and speed” on page 213 9.6.8, “Disk cache write-back versus write-through” on page 214 9.6.9, “RAID adapter cache size” on page 216
Tuning IBM System x Servers for Performance
9.6.10, “Rebuild time” on page 217 9.6.11, “Device drivers and firmware” on page 219 9.6.12, “Fibre Channel performance considerations” on page 220 The topics that we list here are relevant to all disk subsystems, whether these disk subsystems are attached directly or remote. The last section also looks at how to configure the Fibre Channel to achieve the best overall disk performance.
9.6.1 RAID strategy Your RAID strategy should be carefully selected because it significantly affects disk subsystem performance. Figure 9-18 illustrates the relative performance between RAID-0, RAID-1E and RAID-5 using a maximum number of 15 000 RPM drives. The chart shows the RAID-0 configuration delivering about 77% greater throughput than RAID-5 and 43% greater throughput than RAID-1E. RAID-0 has no fault tolerance and is, therefore, best used for read-only environments when downtime for possible backup recovery is acceptable. You need to select RAID-1 or RAID-5 for applications that require fault tolerance. RAID-1E is usually selected when the number of drives is low (less than six) and the price for purchasing additional drives is acceptable. RAID-1E offers about 23% more throughput than RAID-5. You must understand these performance considerations before you select a fault-tolerant RAID strategy.
RAID levels 6000
Configuration: ServeRAID-6i Maximum number of drives 8% capacity (stroke) 15 000 RPM 8 KB I/O size Random I/O Mix: 67/33 R/W
I/Os per second
5000 4000 3000 2000 1000 0
RAID-0
RAID-1E
RAID-5
Figure 9-18 Comparing RAID levels
Chapter 9. Disk subsystem
201
In many cases, RAID-5 is the best choice because it provides the best price and performance combination for configurations requiring capacity greater than five or more disk drives. RAID-5 performance approaches RAID-0 performance for workloads where the frequency of write operations is low. Servers executing applications that require fast read access to data and high availability in the event of a drive failure should employ RAID-5.
9.6.2 Number of drives The number of disk drives affects performance significantly, because each drive contributes to total system throughput. Capacity requirements are often the only consideration used to determine the number of disk drives configured in a server. Throughput requirements are usually not well understood or are completely ignored. Capacity is used because it is estimated easily and is often the only information available. The result is a server configured with sufficient disk space, but insufficient disk performance to keep users working efficiently. High-capacity drives have the lowest price per byte of available storage and are usually selected to reduce total system price. This often results in disappointing performance, particularly if the total number of drives is insufficient. It is difficult to specify server application throughput requirements accurately when attempting to determine the disk subsystem configuration. Disk subsystem throughput measurements are complex. To express a user requirement in terms of “bytes per second” is meaningless because the disk subsystem’s byte throughput changes as the database grows and becomes fragmented and as new applications are added. The best way to understand disk I/O and users’ throughput requirements is to monitor an existing server. You can use tools such as the Windows Performance console to examine the logical drive queue depth and disk transfer rate (described in 14.1, “Performance console” on page 472). Logical drives that have an average queue depth much greater than the number of drives in the array are very busy, which indicates that performance can improved by adding drives to the array. Tip: In general, adding drives is one of the most effective changes that you can make to improve server performance. Measurements show that server throughput for most server application workloads increases as the number of drives configured in the server is increased. As the number of drives is increased, performance usually improves
202
Tuning IBM System x Servers for Performance
for all RAID strategies. Server throughput continues to increase each time drives are added to the server, as shown in Figure 9-19.
Number of drives (RAID-1E)
I/Os per Second
2000
Configuration: ServeRAID-4H 9 GB drives RAID-1E 15 000 RPM 8 KB I/O size Random I/O Mix: 67/33 R/W
1500
1000
500
0
2 drives
4 drives
6 drives
Figure 9-19 Improving performance by adding drives to arrays
This trend continues until another server component becomes the bottleneck. In general, most servers are configured with an insufficient number of disk drives. Therefore, performance increases as drives are added. Similar gains can be expected for all I/O-intensive server applications such as office-application file serving, Lotus Notes, Oracle, DB2, and Microsoft SQL Server. Rule of thumb: For most server workloads, when the number of drives in the active logical array is doubled, server throughput improves by about 50% until other bottlenecks occur. If you are using one of the IBM ServeRAID family of RAID adapters, you can use the logical drive migration feature to add drives to existing arrays without disrupting users or losing data.
9.6.3 Active data set size The active data set is the set of data that an application uses and manipulates on a regular basis. In benchmark measurements, the active data set is often referred to as stroke, as in 10% stroke, meaning that the data is stored on only 10% of the disk surface.
Chapter 9. Disk subsystem
203
As discussed earlier, many drive configurations are based on capacity requirements, which means that the data is stored over a large percentage of the total capacity of the drives. The downside of filling the disks with data is that in most production environments, it translates into reduced performance due to longer seek times. Figure 9-20 illustrates the performance degradation of a disk drive with respect to the size of the active data set.
I/O operations per second per drive
104 90
120
84
81
79
100 80
Workload: OLTP application 8 KB stripe size RAID-0 10 K RPM drives
60 40 20 0
10%
20% 30% 40% % disk stroke
50%
Figure 9-20 Hard drive performance with respect to active data set size
If the active data set spans 50% of the drive capacity, then a single 10K RPM drive is capable of achieving approximately 79 I/Os per second for a simulated database workload that consists of 70% reads and 30% writes that are randomly accessed. Adding enough drives so that the active data set spans only 20% of the drive capacity would increase the drive performance by 14%, to 90 I/Os per second. As shown in Figure 9-20, spreading the active data set across more drives minimizes the seek time and, therefore, improves performance. Disk fragmentation also degrades performance. Over time, files become fragmented on the hard drive, which means that the data in those files is not arranged contiguously on the hard drive. Consequently, a request for a fragmented file will result in multiple seeks in order to satisfy the request. You need to use a disk defragmentation tool on a regular basis to maintain contiguous geometry of data within a file, which can ensure optimized performance.
204
Tuning IBM System x Servers for Performance
9.6.4 Drive performance Drive performance contributes to overall server throughput because faster drives perform disk I/O in less time. There are four major components to the time it takes a disk drive to execute and complete a user request: Command overhead This is the time it takes for the drive’s electronics to process the I/O request. The time depends on whether it is a read or write request and whether the command can be satisfied from the drive’s buffer. This value is of the order of 0.1 ms for a buffer hit to 0.5 ms for a buffer miss. Seek time This is the time it takes to move the drive head from its current cylinder location to the target cylinder. As the radius of the drives has been decreasing, and drive components have become smaller and lighter, so too has the seek time been decreasing. Average seek time is usually 3-5 ms for most current drives used in servers today. Rotational latency When the head is at the target cylinder, the time it takes for the target sector to rotate under the head is called the rotational latency. Average latency is half the time it takes the drive to complete one rotation, so it is inversely proportional to the RPM value of the drive: – – – –
15 000 RPM drives have a 2.0 ms latency 10 000 RPM drives have a 3.0 ms latency 7200 RPM drives have a 4.2 ms latency 5400 RPM drives have a 5.6 ms latency
Data transfer time This value depends on the media data rate, which is how fast data can be transferred from the magnetic recording media, and the interface data rate, which is how fast data can be transferred between the disk drive and disk controller (that is, the SCSI transfer rate). The sum of these two values is typically 1 ms or less. As you can see, the significant values that affect performance are the seek time and the rotational latency. For random I/O (which is normal for a multi-user server), this is true. Reducing the seek time will continue to improve performance as the physical drive attributes become fewer. For sequential I/O (such as with servers with small numbers of users requesting large amounts of data) or for I/O requests of large block sizes (for example 64 KB), the data transfer time does become important when compared to seek
Chapter 9. Disk subsystem
205
and latency, so the use of Ultra320 SCSI versus older SCSI technologies can have a significant positive effect on overall subsystem performance. Likewise, when caching and read-ahead is employed on the drives themselves, the time taken to perform the seek and rotation is eliminated, so the data transfer time becomes very significant. In addition to seek time and rotational latency, current IBM disk drives improve performance by employing advanced I/O command optimization. These drives achieve high performance in part because of a rotational positioning optimization (RPO) scheme. RPO utilizes an onboard microprocessor to sort incoming I/O commands to reduce disk head movement and increase throughput. For example, assuming the disk head is at track number 1 and sector number 1, IBM drives would optimize the following three I/O requests from: 1. Read track 1 sector 2 2. Write track 1 sector 50 3. Read track 3 sector 10 to: 1. Read track 1 sector 2 2. Read track 3 sector 10 3. Write track 1 sector 50 The optimization algorithm of the IBM drives reorders the I/O requests whenever a seek to another track can be accomplished before the disk rotates to the sector of the next I/O request. This technique effectively increases the drive's throughput by processing an I/O command while waiting for the rotational latency of the next I/O request to expire. The easiest way to improve disk performance is to increase the number of accesses that can be made simultaneously by using many drives in a RAID array and spreading the data requests across all drives. See 9.6.2, “Number of drives” on page 202.
9.6.5 Logical drive configuration Using multiple logical drives on a single physical array is convenient for managing the location of different files types. However, depending on the configuration, it can significantly reduce server performance. When you use multiple logical drives, you are physically spreading the data across different sections of the array disks. If I/O is performed to each of the logical drives, the disk heads have to seek further across the disk surface than
206
Tuning IBM System x Servers for Performance
when the data is stored on one logical drive. Using multiple logical drives greatly increases seek time and can slow performance by as much as 25%. An example of this is creating two logical drives in the one RAID array and putting a database on one logical drive and the transaction log on the other. Because heavy I/O is being performed on both, the performance will be poor. If the two logical drives are configured with the operating system on one and data on the other, then there should be little I/O to the operating system code after the server has booted, so this type of configuration would work. It is best to put the page file on the same drive as the data when using one large physical array. This is counterintuitive: most think the page file should be put on the operating system drive, because the operating system will not see much I/O during runtime. However, this causes long seek operations as the head swings over the two partitions. Putting the data and page file on the data array keeps the I/O localized and reduces seek time. Of course, this is not the most optimal case, especially for applications with heavy paging. Ideally, the page drive will be a separate device that can be formatted to the correct stripe size to match paging. In general, most applications will not page when given sufficient RAM, so usually this is not a problem. The fastest configuration is a single logical drive for each physical RAID array. Instead of using logical drives to manage files, you should create directories and store each type of file in a different directory. This will significantly improve disk performance by reducing seek times because the data will be as physically close together as possible. If you really want or need to partition your data and you have a sufficient number of disks, you need to configure multiple RAID arrays instead of configuring multiple logical drives in one RAID array. This improves disk performance; seek time is reduced because the data is physically closer together on each drive. Note: If you plan to use RAID-5EE arrays, you can only have one logical drive per array.
9.6.6 Stripe size With RAID technology, data is striped across an array of hard disk drives. Striping is the process of storing data across all the disk drives that are grouped in an array. The granularity at which data from one file is stored on one drive of the array before subsequent data is stored on the next drive of the array is called the stripe unit (also referred to as interleave depth). For the ServeRAID adapter
Chapter 9. Disk subsystem
207
family, the stripe unit can be set to a stripe unit size of up to 64 KB for older adapters, and up to 1024 KB for newer adapters. With Fibre Channel, a stripe unit is called a segment, and segment sizes can also be up to 1 MB. The collection of these stripe units, from the first drive of the array to the last drive of the array, is called a stripe. Figure 9-21 shows the stripe and stripe unit.
Stripe
SU1 SU4
SU2 SU5
SU3 SU6
Stripe Unit
Figure 9-21 RAID stripes and stripe units
Note: The term stripe size should really be stripe unit size because it refers to the length of the stripe unit (the piece of space on each drive in the array). Using stripes of data balances the I/O requests within the logical drive. On average, each disk will perform an equal number of I/O operations, thereby contributing to overall server throughput. Stripe size has no effect on the total capacity of the logical disk drive.
Selecting the correct stripe size The selection of stripe size affects performance. In general, the stripe size should be at least as large as the median disk I/O request size generated by server applications. Selecting too small a stripe size can reduce performance. In this environment, the server application requests data that is larger than the stripe size, which results in two or more drives being accessed for each I/O request. Ideally, only a single disk I/O occurs for each I/O request. Selecting too large a stripe size can reduce performance because a larger than necessary disk operation might constantly slow each request. This is a problem, particularly with RAID-5 where the complete stripe must be read from disk to calculate a checksum. Use too large a stripe, and extra data must be read each time the checksum is updated.
208
Tuning IBM System x Servers for Performance
Selecting the correct stripe size is a matter of understanding the predominate request size performed by a particular application. Few applications use a single request size for each and every I/O request. Therefore, it is not possible to always have the ideal stripe size. However, there is always a best-compromise stripe size that will result in optimal I/O performance. There are two ways to determine the best stripe size: Use a rule of thumb as per Table 9-4. Monitor the I/O characteristics of an existing server. The first and simplest way to choose a stripe size is to use Table 9-4. This table is based on tests performed by the System x Performance Lab. Table 9-4 Stripe size setting for various applications
Applications
Stripe size
Groupware on Windows (Lotus Domino, Exchange, etc.)
32 KB to 64 KB
Database server on Windows (Oracle, SQL Server, DB2, etc.)
32 KB to 64 KB
File server (Windows)
16 KB
File server (NetWare)
16 KB
File server (Linux)
64 KB
Web server
8 KB
Video file server (streaming media)
64 KB to 1 MB
Other
16 KB
Important: Table 9-4 is only relevant if no information is available with respect to the production environment. The more information that you can determine about the production application and its average data request size, the more accurate the stripe size setting can become. In general, stripe size only needs to be at least as large as the I/O size. Having a smaller stripe size implies multiple physical I/O operations for each logical I/O, which will cause a drop in performance. Using a larger stripe size implies a read-ahead function which might or might not improve performance. Table 9-4 offers rule-of-thumb settings. There is no way to offer the precise stripe size that will always give the best performance for every environment without doing extensive analysis on the specific workload.
Chapter 9. Disk subsystem
209
The second way to determine the correct stripe size involves observing the application while it is running using the Windows Performance console. The key is to determine the average data transfer size being requested by the application and select a stripe size that best matches. Unfortunately, this method requires the system to be running, so it either requires another system running the same application or the reconfiguring of the existing disk array after the measurement has been made (and therefore backup, reformat and restore operations). The Windows Performance console can help you determine the proper stripe size. Select: The object: PhysicalDisk The counter: Avg. Disk Bytes/Transfer The instance: the drive that is receiving the majority of the disk I/O Then, monitor this value. As an example, the trend value for this counter is shown as the thick line in Figure 9-22. The running average is shown as indicated. The figure represents an actual server application. You can see that the application request size (represented by Avg. Disk Bytes/Transfer) varies from a peak of 64 KB to about 20 KB for the two run periods.
210
Tuning IBM System x Servers for Performance
Data drive average disk bytes per transfer: Range of 20 KB to 64 KB Maximum 64 KB
Running Average
Figure 9-22 Average I/O size
Note: This technique is not infallible. It is possible for the Bytes per Transfer counter to have a very high degree of variance. When this occurs, using an average value to select the stripe size is less precise than using a distribution. However, most generally available monitoring software is limited to providing average values for bytes per transfer. Fortunately, using a simple average is sufficient for most applications. As we said at the beginning of this section, in general, the stripe size should be at least as large as the median disk I/O request size generated by the server application. This particular server was configured with an 8 KB stripe size, which produced very poor performance. Increasing the stripe size to 16 KB would improve performance and increasing the stripe size to 32 KB would increase performance even more. The simplest technique would be to place the time window around
Chapter 9. Disk subsystem
211
the run period and select a stripe size that is at least as large as the average size shown in the running average counter. Activating disk performance counters: Windows Server 2003 has both the logical and physical disk counters enabled by default. In Windows 2000, physical disk counters are enabled by default. The logical disk performance counters are disabled by default and might be required for some monitoring applications. If you require the logical counters, you can enable them by typing the command DISKPERF -yv then restarting the computer. Keeping this setting on all the time draws about 2% to 3% CPU, but if your CPU is not a bottleneck, this is irrelevant and can be ignored. Enter DISKPERF /? for more help on the command. In Windows Server 2003, it is even more important to use the Windows System Monitor to determine the average data transfer size. In Windows Server 2003, file data can be cached in 256 KB chunks in the system address space. This allows file data to be read from and written to the disk subsystem in 256 KB data blocks. Therefore, using System Monitor to set the stripe size to match the data transfer size is important. Server applications that serve video and audio are the most common workloads that can have average transfer sizes larger than 64 KB. However, a growing number of real world applications that store video and audio in SQL databases, also have average transfer sizes larger than 64 KB. In cases where the Windows Performance console reports average data transfer sizes that are greater than 64 KB, the stripe unit size should be increased appropriately, if the adapter supports it.
Page file drive Windows Server 2003 performs page transfers at up to 64 KB per operation, so the paging drive stripe size can be as large as 64 KB. However, in practice, it is usually closer to 32 KB because the application might not make demands for large blocks of memory which limits the size of the paging I/O. Monitor average bytes per transfer as described in “Selecting the correct stripe size” on page 208. Setting the stripe size to this average size can result in a significant increase in performance by reducing the amount of physical disk I/O that occurs because of paging.
212
Tuning IBM System x Servers for Performance
For example, if the stripe size is 8 KB and the page manager is doing 32 KB I/O transfers, then four physical disk reads or writes must occur for each page per second that you see in the Performance console. If the system is paging 10 pages per second, then the disk is actually doing 40 disk transfers per second.
9.6.7 SCSI bus organization and speed Concern often exists over the performance effects caused by the number of drives on the SCSI bus, or the speed at which the SCSI bus runs. Yet, in almost all modern server configurations, the SCSI bus is rarely the bottleneck. In most cases, optimal performance can be obtained by simply configuring 14 drives per Ultra SCSI bus. If the application is byte-I/O-intensive (as is the case with video or audio), and is sustaining throughputs approaching 75% of the SCSI buses theoretical maximum, reducing the number of drives on each SCSI bus can yield a moderate increase in system performance. In general, it is rare that SCSI bus configuration or increasing SCSI bus speed significantly improves overall server system performance. Consider that servers must access data stored on disk for each of the attached users. Each user is requesting access to different data stored in a unique location on the disk drives. Disk accesses are almost always random because the server must multiplex access to disk data for each user. This means that most server disk accesses require a seek and rotational latency before data is transferred across the SCSI bus.
SCSI bus speed As described in 9.6.4, “Drive performance” on page 205, total disk seek and latency times average about 6-8 ms, depending on the speed of the drive. Transferring a 4 KB block over a 320 MBps SCSI bus takes about 0.12 ms, or approximately 1/50th of the total disk access time, a very small fractional gain in overall performance. Tests have shown that for random I/O, drive throughput usually does not approach the limits of the SCSI bus. File-serving and e-mail benchmarks (and applications) transfer relatively large blocks of data (12 KB to 64 KB), which increases SCSI bus utilization. More importantly, however, these benchmarks usually build a relatively small set of data files, resulting in artificially reduced disk seek times. In production environments, disk drives are usually filled to at least 30% to 50% of their capacity, causing longer seek times compared to the benchmark files that might only use 1% to 2% of the disk capacity. After all, building a 2 GB database for a benchmark might seem like a large data set, but on a disk array containing five 36 GB drives, that database uses less than 1/90th of the total space. Using only
Chapter 9. Disk subsystem
213
a small fraction of the available disk space greatly reduces seek times, thereby inflating the performance contribution of the SCSI bus.
PCI bus Do not forget that all this data must travel though the PCI bus. In the past, PCI bus performance prevented the maximum SCSI throughput from being achieved. Due to the faster PCI to memory performance of current servers all current ServeRAID products only have two SCSI channels. Specification-driven technologies, such as SCSI, are often motivated by desktop environments. In the desktop environment, applications tend to be more sequential, and the system usually has a single SCSI adapter that can monopolize much of the PCI to memory bandwidth. In the desktop environment, Ultra320 SCSI provides significant performance gains. Because of the more random nature of server applications, these benefits often do not translate to server environments.
Multiple SCSI buses The SCSI bus organization of drives on a multi-bus controller (such as ServeRAID) does not significantly affect performance for most server workloads. For example, in a four-drive configuration, it does not matter whether you attach all drives to a single SCSI bus or two drives each to two different SCSI buses. In applications such as database transaction processing, both configurations usually have identical disk subsystem performance. The SCSI bus does not contribute significantly to the total time required for each I/O operation. Each I/O operation usually requires drive seek and latency times; therefore, the sustainable number of operations per second is reduced, causing SCSI bus utilization to be low. For a configuration which runs applications that access image data or large sequential files, performance improvement can be achieved by using a balanced distribution of drives on the SCSI buses of the ServeRAID.
9.6.8 Disk cache write-back versus write-through Most people think that write-back mode is always faster because it allows data to be written to the disk controller cache without waiting for disk I/O to complete. This is usually the case when the server is lightly loaded. However, as the server becomes busy, the cache fills completely, causing data writes to wait for space in the cache before being written to the disk. When this happens, data write operations slow to the speed at which the disk drives empty the cache. If the server remains busy, the cache is flooded by write requests, which results in a
214
Tuning IBM System x Servers for Performance
bottleneck. This bottleneck happens regardless of the size of the adapter’s cache. In write-through mode, write operations do not wait in cache memory that must be managed by the processor on the RAID adapter. When the server is lightly loaded (the left side of the figure in Figure 9-23), write operations take longer because they cannot be quickly stored in the cache. Instead, they must wait for the actual disk operation to complete. Thus, when the server is lightly loaded, throughput in write-through mode is generally lower than in write-back mode. 2000
Write-through
I/Os per second
1500
Write-back
Comparing write-through versus write-back ServeRAID-7k 8 KB random I/O Mix: 67/33 R/W
1000
500
0 Increasing load
Figure 9-23 Comparing write-through and write-back modes under increasing load
However, when the server becomes very busy (the right side of the figure in Figure 9-23), I/O operations do not have to wait for available cache memory. I/O operations go straight to disk, and throughput is usually greater for write-through than in write-back mode. Write-through is also faster when battery-backup cache is installed; this is due partly to the fact that the cache is mirrored. Data in the primary cache has to be copied to the memory on the battery-backup cache card. This copy operation eliminates a single point of failure, thereby increasing the reliability of the controller in write-back mode, but this takes time and slows writes, especially when the workload floods the adapter with write operations. The difficult part is to determine where this crossover point is. There is no set rule of thumb because each server and corresponding workload is different. However, as a starting point, use Performance Monitor to determine whether the counter Average Disk sec/write (write response time) is greater than 40 ms. If
Chapter 9. Disk subsystem
215
this is the case and more drives cannot be added to reduce the response time, we recommend that you use the write-though cache policy. Rules of thumb: Based on Figure 9-23, the following rules of thumb is appropriate. If the disk subsystem is very busy (write response time greater than 40 ms), use write-through mode. If the disks are configured correctly, and the server is not heavily loaded, use write-back mode.
9.6.9 RAID adapter cache size Tests show that the ServeRAID-6M adapter with 256 MB of cache does not typically outperform the 128 MB version of the same card for most real-world application workloads. After the cache size is above the minimum required for the job, the extra cache usually offers little additional performance benefit. The cache increases performance by providing data that would otherwise be accessed from disk. However, in real-world applications, total data space is so much larger than disk cache size that, for random operations, there is very little statistical chance of finding the requested data in the cache. For example, a 200 GB database would not be considered very large by today's standards. A typical database of this size might be placed on an array consisting of seven or more 36 GB drives. For random accesses to such a database, the probability of finding a record in the cache would be the ratio of 128 MB:200 GB or approximately 1 in 1600 operations. Double the cache size and this value is decreased by half, which is still a very discouraging hit-rate. It would take a very large cache to increase the cache hit-rate to the point where caching becomes advantageous for random accesses. In RAID-5 mode, significant performance gains from write-back mode are derived from the ability of the disk controller to merge multiple write commands into a single disk write operation. In RAID-5 mode, the controller must update the checksum information for each data update. Write-back mode allows the disk controller to keep the checksum data in adapter cache and perform multiple updates before completing the update to the checksum information contained on the disk. In addition, this does not require a large amount of RAM. In most cases, disk array caches can usually provide high hit rates only when I/O requests are sequential. In this case, the controller can pre-fetch data into the cache so that on the next sequential I/O request, a cache hit occurs. Pre-fetching for sequential I/O requires only enough buffer space or cache memory to stay a
216
Tuning IBM System x Servers for Performance
few steps ahead of the sequential I/O requests. This can be done with a small circular buffer. Having a large cache often means more memory to manage when the workload is heavy; during light loads, very little cache memory is required. Most people do not invest the time to think about how cache works. Without much thought, it is easy to reach the conclusion that “bigger is always better.” The drawback is that larger caches take longer to search and manage. This can slow I/O performance, especially for random operations, because there is a very low probability of finding data in the cache. Benchmarks often do not reflect a customer production environment. In general, most “retail” benchmark results run with very low amounts of data stored on the disk drives. In these environments, a very large cache will have a high hit-rate that is artificially inflated compared to the hit-rate from a production workload. In a production environment, an overly large cache can actually slow performance as the adapter continuously searches the cache for data that is never found, before it starts the required disk I/O. This is the reason that many array controllers turn off the cache when the hit-rates fall below an acceptable threshold. In identical hardware configurations, it takes more CPU overhead to manage 128 MB of cache compared to 64 MB and even more for 256 MB. The point is that bigger caches do not always translate to better performance. Although ServeRAID-6M has a 600 MHz Intel Verde I/O processor, this CPU is approximately four times faster than the 100 MHz Intel Zion I/O processor used on ServeRAID-4Mx. Therefore, ServeRAID-6M can manage the larger cache without running more slowly than ServeRAID-4Mx. Furthermore, the amount of cache must be proportional to the number of drives attached. Typically, cache hits are generated from sequential read ahead. You do not need to read ahead very much to have 100% hits.
9.6.10 Rebuild time Rebuild time is an important part of the overall performance of a RAID subsystem. The longer the disk subsystem spends recovering from a drive failure, the longer the subsystem is vulnerable to losing data if another drive failure occurs. In addition, the performance of a RAID controller that is busy doing a rebuild will also be less. Rebuild time varies depending on the RAID controller, the capacity of the drive, the number of drives in an array, the I/O workload during rebuild and the RAID
Chapter 9. Disk subsystem
217
level. Obviously, the smaller the drive, the shorter the rebuild time, so the rebuild time of the various RAID levels will be the focus of this section. On current ServeRAID adapters, the RAID levels that provide fault tolerance are RAID 1, RAID-10, RAID-1E, RAID-5, RAID-50, and RAID-5EE. In general, RAID-1 arrays have the fastest rebuild time because it only requires data from one other drive to rebuild. RAID-10 and RAID-1 Because RAID-10 is the result of forming a RAID-0 array from two or more RAID-1 arrays, RAID-10 rebuild times are similar to RAID-1 rebuild times. RAID-1E RAID-1E requires data from at least two other drives to rebuild, so it takes longer to rebuild than a RAID-1 or RAID-10 array (Figure 9-24).
Rebuild time
Rebuild time (minutes)
50
47
40
Configuration: ServeRAID-6i Four drives 36 GB size 15 K RPM 100% stroke
30 25
20 16
10
10
0 RAID-10
RAID-1E
RAID-5
RAID-5EE
Figure 9-24 Rebuild time comparison between different RAID levels
RAID-5 and RAID-50 To rebuild a RAID-5 array, each data stripe on the new drive must be reconstructed from the data stripes and checksums (also known as a parity stripes) from the other drives in the array. This process can take much longer than a RAID-10 rebuild. Because RAID-50 is the result of forming a RAID-0 array from two or more RAID-5 arrays, RAID-50 rebuild times are similar to RAID-5 rebuild times.
218
Tuning IBM System x Servers for Performance
RAID-5EE RAID-5EE provides much faster rebuild times than RAID 5E (Figure 9-25), but its rebuild time is still slower than other RAID levels. The RAID-5E level should be replaced by RAID-5EE wherever possible.
Rebuild time 15 Configuration: ServeRAID-4Mx Four drives Size 36 GB 15 K RPM 100% Stroke
Time (hours)
12.5
10
5
1
0 RAID-5EE
RAID-5E
Figure 9-25 Rebuild time - RAID 5E versus RAID 5EE
9.6.11 Device drivers and firmware Device drivers play a major role in performance of the subsystem with which the driver is associated. A device driver is software written to recognize a specific device. Most of the device drivers are vendor-specific; these drivers are supplied by the hardware vendor (such as IBM in the case of ServeRAID). For IBM ServeRAID and Fibre Channel products, go to the IBM support site to download the latest version: http://www.ibm.com/servers/eserver/support/xseries/ The same applies to firmware. The firmware is stored on the disk controller itself (for example, the ServeRAID adapter) and is often the source of significant performance improvements. Firmware updates are also available from the above site but you should review the Read Me file associated with the upgrade to determine if performance improvements are likely. Wherever it is practical, we recommend that you always maintain your servers with the latest version of driver and firmware. It might not be practical if your system requires a high level of uptime and there is currently no performance or
Chapter 9. Disk subsystem
219
access problem (in other words, “if it isn’t broken, don’t fix it”). Specifically for ServeRAID, we recommend that you ensure that the driver, firmware, and ServeRAID Manager code are always at the same level. It should also be noted that often, the latest driver is not the best or correct driver to use. This note is especially important with specific hardware configurations that are certified by an application vendor. An example of this circumstance is Microsoft’s Cluster Server. You must check the certified configuration to determine what driver level is supported.
9.6.12 Fibre Channel performance considerations Let us look at what happens when a read I/O operation is requested to a Fibre Channel subsystem, and the data requested is not located in the RAID controller disk cache: 1. A read command is generated by the server and the read command includes the logical block address of the data being requested. 2. The SCSI command is encapsulated by Fibre Channel frames and transmitted by the Fibre Channel host adapter to the RAID controller over the Fibre Channel link. 3. The RAID controller parses the read command and uses the logical block address to issue the disk read command to the correct drive. 4. The disk drive performs the read operation and returns the data to the RAID controller. 5. The Fibre Channel electronics within the RAID controller encapsulate the data Fibre Channel frames. The data is transferred to the server over the Fibre Channel link. 6. When in the Fibre Channel adapter, the data is transferred over the PCI bus into memory of the server. Of course, a large amount of the detail was left out, but this level of observation is sufficient to understand the most important performance implication of Fibre Channel. The Fibre Channel link, like most network connections, sustains a data transfer rate that is largely determined by the payload of the frame. Stated another way, the throughput of Fibre Channel is a function of the disk I/O size being transferred. This is because Fibre Channel frames have a maximum data payload of 2112 bytes. Data transfers for larger data sizes require multiple Fibre Channel frames.
220
Tuning IBM System x Servers for Performance
Figure 9-26 illustrates the effects of disk request size on Fibre Channel throughput. At small disk request sizes such as 2 KB, the maximum Fibre Channel throughput is about 225 MBps or about 30% the maximum achievable bandwidth that four 2 Gbps links provide. These values provide critical information, because many people think the maximum bandwidth of a Fibre Channel link or the maximum aggregate bandwidth of multiple links is obtained for all operations.
Fibre Channel througput versus transfer request size DS4500 with four 2 Gbps links 800 Transfer rate (MBps)
700 600 500 400 300 200 100 0 0.5
1
2
4
8
16
32
64
128
Transfer request size (KB)
256
512
Overhead Throughput
Figure 9-26 Fibre Channel throughput versus transfer request size
Only when the transfer request size is as large as 64 KB does Fibre Channel begin to reach its maximum sustainable transfer rate over four 2 Gbps links. In this case, the maximum transfer rate is approximately 750 MBps. Why is the maximum transfer rate of four 2 Gbps links not equal to 800 MBps (4 x 2 Gbit links equals 800 MBps taking into account 2-bit serial overhead for every byte)? The maximum theoretical throughput for a 1 Gbps Fibre Channel link was 92 MBps, which is 92% of the maximum theoretical throughput of a 1 Gbit link (100 MBps). It is interesting to note that the maximum throughput of four 2 Gbps links (750 MBps) is approximately 94% of the maximum theoretical throughput of four 2 Gbit links (800 MBps). The overhead has remained nearly constant from first to second generation Fibre Channel links.
Chapter 9. Disk subsystem
221
The difference between the measured result and the theoretical maximum throughput can be explained by overhead of command and control bits that accompany each Fibre Channel frame. This is discussed in the following sections.
Fibre Channel protocol layers We can get a better appreciation for the overhead described in the previous section if we take a brief look at the Fibre Channel layers and the Fibre Channel frame composition. The Fibre Channel specification defines five independent protocol layers (Figure 9-27). These layers are structured so that each layer has a specific function to enable reliable communications for all of the protocols supported by Fibre Channel standard.
SCSI
HiPPi
IPI
SBCCS IP FC-4 Mapping Protocol
802.2
FC-3 Common Services Protocol FC-2 Signaling and Framing Protocol FC-1 Transmission Protocol FC-0 Physical Figure 9-27 Fibre Channel functional levels
The five independent layers are: FC-0 is the physical layer. It is comprised of the actual wire or optical fibre over which data travels. FC-1 is the transmission protocol. The Transmission layer is responsible for encoding of the bits on the physical medium, for data transmission error detection, and for signal clock generation. FC-2 is important from a performance perspective because this is the layer that is responsible for building the data frames that flow over the Fibre Channel link. FC-2 is also responsible for segmenting large transfer requests into multiple Fibre Channel frames. FC-3 defines the common services layer. This layer is responsible for defining the common services that are accessible across all Fibre Channel ports. One
222
Tuning IBM System x Servers for Performance
of these services is the Name Server. The Name Server provides a directory of all the Fibre Channel nodes accessible on the connection. For example, a Fibre Channel switch would be a name server and maintain a directory of all the ports attached to that switch. Other Fibre Channel nodes could query the switch to determine what node addresses are accessible through that switch. FC-4 defines the protocol standards that can be used to transport data over Fibre Channel. Some of these protocols include: – – – – – –
SCSI (Small Computer Systems Interface) HiPPI (High Performance Parallel Interface) IPI (Intelligent Peripheral Interface) SBCCS (Single Byte Command Code Set) to support ESCON® IP (Internet Protocol) 802.2
Our discussion is limited to SCSI because the IBM DS4000 RAID controller products are based upon the SCSI protocol. Fibre Channel allows the SCSI protocol commands to be encapsulated and transmitted over Fibre Channel to SCSI devices connected to the RAID controller unit. This is significant because this technique allows Fibre Channel to be quickly developed and function with existing SCSI devices and software.
The importance of the I/O size Considering the shape of the throughput chart in Figure 9-26 on page 221, we can deduce that the throughput of Fibre Channel is clearly sensitive to the disk access size. Small disk access sizes have low throughput while larger blocks have greater overall throughput. The reason for this can be seen by looking at the read command example that we discussed in 9.6.12, “Fibre Channel performance considerations” on page 220. In the case of a 2 KB read operation, the sequence is: 1. A SCSI read command is issued by the device driver to the Fibre Channel host adapter at level FC-4. 2. On the host side, the SCSI read command must flow down from FC-4 to FC-0 before it is transferred over the Fibre Channel link to the external RAID controller. 3. The RAID controller also has a Fibre Channel interface that receives the read command at FC-0 and sends it up through FC-1, FC-2, FC-3, to the SCSI layer at FC-4. 4. The SCSI layer then sends the read command to the Fibre Channel RAID controller. 5. The SCSI read command is issued to the correct disk drive.
Chapter 9. Disk subsystem
223
6. When the read operation completes, data is transferred from the drive to SCSI layer FC-4 of the Fibre Channel interface within the RAID controller. 7. Now the read data must make the return trip down layers FC-4, FC-3, FC-2, FC-1 on the RAID controller side and onto the Fibre Channel link. 8. When the data arrives on the Fibre Channel link, it is transmitted to the host adapter in the server. 9. Again it must travel up the layers to FC-4 on the server side before the SCSI device driver responds with data to the requesting process. Contrast the 2 KB read command with a 64 KB read command and the answer becomes clear. Like the 2 KB read command, the 64 KB read command travels down FC-4, FC-3, FC-2, and to FC-1 on the server side. It also travels up the same layers on the RAID controller side. However, here is where things are different. After the 64 KB read command completes, the data is sent to FC-4 of the Fibre Channel interface on the RAID controller side. The 64 KB data travels down from FC-4, FC-3 and to FC-2. At layer FC-2, the 64 KB data is formatted into a 2112-byte payload to be sent over the link. But 64 KB do not fit into a 2112-byte payload. Therefore, layer FC-2 performs segmentation and breaks the 64 KB disk data up into 32 separate Fibre Channel frames to be sent to the IBM DS4000 controller. Of the 32 frames, 31 frames never had to traverse layers FC-4 and FC3 on the RAID controller side. Furthermore, 31 of these frames never required a separate read command to be generated at all. They were transmitted with one read command. Thus, reading data in large blocks introduces significant efficiencies because much of the protocol overhead is reduced. Any transfer exceeding the 2112 byte payload is shipped as “low-cost” frames back to the host. This knowledge explains why throughput at smaller frame sizes (Figure 9-26 on page 221) is so low and throughput for larger frames improves as the disk I/O size increases. The overhead of the FC-4, FC-3 layers and the additional SCSI read or write commands slow throughput.
Configuring Fibre Channel for performance The important thing to understand is that degradation of throughput with smaller I/O sizes occurs. You must use that information to better configure your Fibre Channel configuration.
224
Tuning IBM System x Servers for Performance
One way to improve performance is to profile an existing server to get an idea of the average disk transfer size by using the Performance console and by examining the following physical disk counters: Average disk bytes/transfer This counter can be graphed versus time to tell you the predominant transfer size for the particular application. This value can be compared to Figure 9-26 on page 221 to determine the maximum level of throughput a single Fibre Channel link can sustain for a particular application. Disk bytes/second This counter tells you what the current disk subsystem is able to sustain for this particular application. This value can also be compared to the maximum throughput obtained from Figure 9-26 on page 221 to determine whether multiple links should be used to reach the target level of throughput demanded for the target number of users. As well as adding a PCI host adapter, you can improve performance by adding a second controller module to the DS4000 unit. Throughput nearly doubles for all transfer sizes when a second controller is added to a DS4500 system, as shown in Figure 9-28.
DS4000 Single Controller versus Dual Controller 800
Transfer rate (MBps)
700
IBM DS4500 Dual Controller
600 500 400 300 200
IBM DS4500 Single Controller
100 0 0.5
1
2
4
8
16
32
64
128
256
512
Transfer request size (KB)
Figure 9-28 Comparing single versus dual controller throughputs
Chapter 9. Disk subsystem
225
Rules of thumb: Double the number of users requires double the amount of disk I/O. Use Figure 9-26 on page 221 to determine the maximum sustainable throughput. If your expected throughput exceeds this value, add another RAID controller. Using a second RAID controller blade doubles the throughput, provided no other bottleneck is created in the process. The remainder of the challenges of optimizing Fibre Channel are similar to those present when configuring a standard RAID controller. Disk layout and organization, such as RAID strategy, stripe size and the number of disks, all affect performance of the IBM Fibre Channel RAID controller in much the same way that it does for ServeRAID. You can use the same techniques that you used to determine these settings for ServeRAID to optimize the IBM Fibre Channel RAID controller solution. Using a large number of drives in an array is the best way to increase throughput for applications that have high I/O demands. These applications include database transaction processing, decision support, e-commerce, video serving, and groupware such as Lotus Notes and Microsoft Exchange. The IBM DS4000 product portfolio includes the DS4100, the DS4300, the DS4300 Turbo, the DS4500, and the DS4800. The DS4100 offers SATA technology for the drives. It is designed for nearline storage applications and data streaming applications. The DS4800 is the latest product in the DS4000 series and can sustain large block sequential transfer rates above 1500 MBps.
Comparing Fibre Channel with SATA The SATA-based DS4100 is suited primarily for sequential I/O operations. In this environment, the DS4100 performs very well. Figure 9-29 illustrates the throughput obtained when performing 100% sequential reads (such as from a video streaming application). Note that the DS4500 is the best performer for streaming workloads and that the SATA-based DS4100 is as good as (if not better than) the DS4300. The significant difference in maximum throughput capability can be attributed to the RAID controllers. The DS4500 is a more powerful RAID controller than the DS4300. The similar performance in maximum throughput capability between the DS4300 and the DS4100 can be attributed to the fact that both of these RAID controllers are very similar with the exception that the DS4100 is a storage server based on SATA, and the DS4300 is a storage server based on Fibre Channel.
226
Tuning IBM System x Servers for Performance
DS4000 maximum throughput 100% sequential read
700
Configuration: 64 KB segment Read cache enabled 16 KB cache block 8% drive stroke Dual RAID controllers RAID-5 14 drives DS4300: 15 K RPM DS4100: 7200 RPM
DS4500 (FC)
600 500
DS4300 (FC)
400 300 200
512
256
128
64
16
8
4
2
0.5
0
32
DS4100 (SATA)
100 1
Transfer rate (MBps)
800
Transfer request size (KB)
Figure 9-29 DS4000 throughput comparison using 100% sequential read operations
For random OLTP-type workloads, however, the DS4300 can handle significantly more I/O requests than the DS4100, as shown in Figure 9-30.
DS4000 I/O operations per second OLTP random workload 70-30 read-write mix
I/O operations per second
3000 2500
Configuration: 64 KB segment No cache enabled 8 KB transfer size 8% drive stroke Dual RAID controllers RAID-5 14 drives DS4300: 15 K RPM DS4100: 7200 RPM
DS4300 (FC)
2000 1500 1000
DS4100 (SATA)
500 0 0
10
20
30
40
50
I/O request queue depth
Figure 9-30 DS4000 throughput using 70-30 read-write mix
The data illustrated in Figure 9-30 substantiates the fact that the D4100 is designed for seldom accessed near-line storage. It should not be implemented as part of a storage subsystem used for accessing data frequently, such as file serving, Web serving, database applications, and mail applications. SATA drive
Chapter 9. Disk subsystem
227
technology is currently not robust enough to handle the duty cycles of enterprise class applications. For more detailed DS4000 performance information, read the following papers written by Charles Stephan of the System x Performance Lab: Comparison of the Performance of the DS4300 (FAStT600) and FAStT200 Storage Servers Comparison of the Performance of the DS4500 (FAStT900) and DS4400 (FAStT700) Storage Servers These papers are both available in PDF format from: http://www.ibm.com/servers/eserver/xseries/benchmarks/related.html
9.7 Disk subsystem rules of thumb A performance relationship can be developed for the disk subsystem. This relationship is based upon the RAID strategy, number of drives, and the disk drive model. Table 9-5 states the disk subsystem rules of thumb. Table 9-5 Disk subsystem rules of thumb
228
Performance of this configuration
Is equivalent to...
RAID-0
33% to 50% more throughput than RAID-1 (same number of drives)
RAID-1E
33% to 50% more throughput than RAID-5 (same number of drives)
RAID-5E
10% to 20% more throughput than RAID-5
RAID-5
30% to 50% more throughput than RAID-6
Doubling number of drives
50% increase in drive throughput (until disk controller becomes a bottleneck)
One 10,000 RPM drive
10% to 50% improvement over 7200 RPM drives (50% when considering RPM only, 10% when comparing with 7200 RPM drives with rotational positioning optimization)
One 15,000 RPM drive
10% to 50% improvement over 10,000 RPM drives
Ultra160 SCSI
5% to 10% more throughput than Ultra SCSI for typical server environments
Tuning IBM System x Servers for Performance
Performance of this configuration
Is equivalent to...
Ultra320 SCSI
5% to 10% more throughput than Ultra160 SCSI for typical server environments
9.8 Tuning with IBM DS4000 Storage Manager IBM DS4000 Storage Manager is the software that lets you manage the DS4000 RAID controllers. It includes its own performance monitoring tool, the Subsystem Management Performance Monitor, which gives you information about the performance aspects of your Fibre Channel subsystem. Figure 9-31 shows the Performance Monitor window of the DS4000 Storage Manager tool. Note: This performance monitor tool is not related to the Windows Performance console tool.
Figure 9-31 Subsystem Management Performance Monitor
This section describes how to use data from the Subsystem Management Performance Monitor and what tuning options are available in the Storage Manager for optimizing the Fibre Channel subsystem’s performance. You use the Subsystem Management Performance Monitor to monitor storage subsystem performance in real time and save performance data to a file for later analysis. You can specify the logical drives and controllers to monitor and the polling interval. Also, you can receive storage subsystem totals, which is data that combines the statistics for both controllers in an active-active controller pair.
Chapter 9. Disk subsystem
229
Table 9-6 describes the data that is displayed for selected devices. Table 9-6 Subsystem management performance monitor parameters
Data field
Description
Total I/Os
Total I/Os performed by this device since the beginning of the polling session. For more information, see “Balancing the I/O load” on page 230.
Read percentage
The percentage of total I/Os that are read operations for this device. Write percentage can be calculated as 100 minus this value. For more information, see “Optimizing the I/O request rate” on page 231.
Cache hit percentage
The percentage of reads that are processed with data from the cache rather than requiring a read from disk. For more information, see “Optimizing the I/O request rate” on page 231.
Current KBps
Average transfer rate during the polling session. The transfer rate is the amount of data in Kilobytes that can be moved through the I/O Data connection in a second (also called throughput). For more information, see “Optimizing the transfer rate” on page 231.
Maximum KBps
The maximum transfer rate that was achieved during the Performance Monitor polling session. For more information, see “Optimizing the transfer rate” on page 231.
Current I/O per second
The average number of I/O requests serviced per second during the current polling interval (also called an I/O request rate). For more information, “Optimizing the I/O request rate” on page 231.
Maximum I/O per second
The maximum number of I/O requests serviced during a one-second interval over the entire polling session. For more information, see “Optimizing the I/O request rate” on page 231.
Balancing the I/O load The Total I/O data field is useful for monitoring the I/O activity to a specific controller and a specific logical drive. This field helps you identify possible I/O hot spots. Identify actual I/O patterns to the individual logical drives and compare those with the expectations based on the application. If a particular controller has considerably more I/O activity than expected, consider moving an array to the other controller in the storage subsystem using the Array → Change Ownership option. Because I/O loads are constantly changing, it can be difficult to perfectly balance I/O load across controllers and logical drives. The logical drives and data accessed during your polling session depend on which applications and users were active during that time period. It is important to monitor performance during different time periods and gather data at regular intervals so you can identify
230
Tuning IBM System x Servers for Performance
performance trends. The Performance Monitor tool allows you to save data to a comma-delimited file so you can import it to a spreadsheet for further analysis. If you notice that the workload across the storage subsystem (Storage Subsystem Totals Total I/O statistic) continues to increase over time while application performance decreases, this can indicate the need to add additional storage subsystems to your enterprise. By doing this, you can continue to meet application needs at an acceptable performance level.
Optimizing the transfer rate As described in 9.6.12, “Fibre Channel performance considerations” on page 220, the transfer rates of the controller are determined by the application I/O size and the I/O request rate. In general, a small application I/O request size results in a lower transfer rate but provides a faster I/O request rate and a shorter response time. With larger application I/O request sizes, higher throughput rates are possible. Understanding your typical application I/O patterns can give you an idea of the maximum I/O transfer rates that are possible for a given storage subsystem. Because of the dependency on I/O size and transmission media, the only technique you can use to improve transfer rates is to improve the I/O request rate. Use the Windows Performance console to gather I/O size data so you understand the maximum transfer rates possible. Then use tuning options available in Storage Manager to optimize the I/O request rate so you can reach the maximum possible transfer rate.
Optimizing the I/O request rate The factors that affect the I/O request rate include:
I/O access pattern (random or sequential) and I/O size Whether write caching is enabled Cache hit percentage RAID level Segment size Number of drives in the arrays or storage subsystem Fragmentation of files Logical drive modification priority Note: Fragmentation affects logical drives with sequential I/O access patterns, not random I/O access patterns.
To determine if your I/O has sequential characteristics, try enabling a conservative cache read-ahead multiplier (4, for example) using the Logical drive → Properties option. Then examine the logical drive cache hit percentage
Chapter 9. Disk subsystem
231
to see if it has improved. An improvement indicates your I/O has a sequential pattern. Use the Windows Performance console to determine the typical I/O size for a logical drive. Higher write I/O rates are experienced with write-caching enabled compared to disabled, especially for sequential I/O access patterns. Regardless of your I/O pattern, it is recommended that you enable write-caching to maximize I/O rate and shorten application response time.
Optimizing the cache hit percentage A higher cache hit percentage is also desirable for optimal application performance and is positively correlated with I/O request rate. If the cache hit percentage of all logical drives is low or trending downward, and you do not have the maximum amount of controller cache memory installed, this could indicate the need to install more memory. If an individual logical drive is experiencing a low cache hit percentage, consider enabling cache read-ahead (or prefetch) for that logical drive. Cache read-ahead can increase the cache hit percentage for a sequential I/O workload. If cache read-ahead is enabled, the cache reads the data from the disk. However, in addition to the requested data, the cache also fetches more data, usually from adjacent data blocks on the drive. This feature increases the chance that a future request for data could be fulfilled from the cache rather than requiring disk access. The cache read-ahead multiplier values specify the multiplier to use for determining how many additional data blocks are read into cache. Choosing a higher cache read-ahead multiplier can increase the cache hit percentage. If you have determined that your I/O has sequential characteristics, try enabling an aggressive cache read-ahead multiplier (8, for example) using the Logical drive → Properties option. Then examine the logical drive cache hit percentage to see if it has improved. Continue to customize logical drive cache read-ahead to arrive at the optimal multiplier (in the case of a random I/O pattern, the optimal multiplier is zero).
Choosing an appropriate RAID level Use the read percentage for a logical drive to determine actual application behavior. Applications with a high read percentage do very well using RAID-5 logical drives because of the outstanding read performance of the RAID-5 configuration.
232
Tuning IBM System x Servers for Performance
However, applications with a low read percentage (write-intensive) do not perform as well on RAID-5 logical drives because of the way a controller writes data and redundancy data to the drives in a RAID-5 array. If there is a low percentage of read activity relative to write activity, you might consider changing the RAID level of an array from RAID-5 to RAID-1 for faster performance.
Choosing an optimal logical drive modification priority The modification priority defines how much processing time is allocated for logical drive modification operations versus system performance. The higher the priority, the faster logical drive modification operations complete but the slower system I/O is serviced. Logical drive modification operations include reconstruction, copyback, initialization, media scan, defragmentation, change of RAID level, and change of segment size. The modification priority is set for each logical drive using a slider bar on the Logical drive → Properties dialog box. There are five relative settings on the reconstruction rate slider bar ranging from Low to Highest. The actual speed of each setting is determined by the controller. Choose the Low setting to maximize the I/O request rate. If the controller is idle (not servicing any I/O), it ignores the individual logical drive rate settings and processes logical drive modification operations as fast as possible.
Choosing an optimal segment size A segment is the amount of data, in kilobytes, that the controller writes on a single drive in a logical drive before writing data on the next drive. With ServeRAID, this is the stripe unit size or stripe size. Data blocks store 512 bytes of data and are the smallest units of storage. The size of a segment determines how many data blocks it contains. For example, an 8 KB segment holds 16 data blocks and a 64 KB segment holds 128 data blocks. Note: The segment size was expressed in number of data blocks in previous versions of this storage management software. It is now expressed in KB. When you create a logical drive, the default segment size is a good choice for the expected logical drive usage. You can change the default segment size using the Logical drive → Change Segment Size option. If your typical I/O size is larger than your segment size, increase your segment size in order to minimize the number of drives needed to satisfy an I/O request. If you are using the logical drive in a single-user, large I/O environment such as multimedia application storage, performance is optimized when a single I/O
Chapter 9. Disk subsystem
233
request can be serviced with a single array data stripe (the segment size multiplied by the number of drives in the array used for I/O). In this case, multiple disks are used for the same request, but each disk is only accessed once.
Minimizing disk accesses by defragmentation Each access of the drive to read or write a file results in spinning of the drive platters and movement of the read/write heads. Make sure the files on your array are defragmented. When the files are defragmented, the data blocks that makes up the files are next to each other so that the read/write heads do not have to travel all over the disk to retrieve the separate parts of the file. Fragmented files are detrimental to the performance of a logical drive with sequential I/O access patterns.
234
Tuning IBM System x Servers for Performance
10
Chapter 10.
Network subsystem Because all server applications provide services to users who are connected through a network, the network subsystem and the network itself play a crucial role in server performance from the user’s point of view. This chapter covers the following topics:
10.1, “LAN operations” on page 236 10.2, “Factors affecting network controller performance” on page 242 10.3, “Advanced network features” on page 259 10.4, “Internet SCSI (iSCSI)” on page 283 10.5, “Interconnects” on page 289 Note: Throughout this book, B represents bytes and b represents bits: MB, MBps, KB, KBps refer to bytes (megabytes per second, for example) Mb, Gb, Mbps, Kbps refer to bits (megabits per second, for example)
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
235
10.1 LAN operations Ethernet has emerged as the dominant networking technology in most of today’s local area networks (LANs). Competing technologies such as token-ring, FDDI and asynchronous transfer mode (ATM) are no longer widely used for LANs. ATM is still a prevalent choice for running high-bandwidth networks that span large areas. With current transfer rates of more than 10 Gbps, a lot of Internet service providers and telecommunications companies have implemented ATM as their backbone technology. Although Ethernet is in some respects technically inferior to other networking technologies, its low cost, ease of use and backward-compatibility make it a viable choice for most LANs today. Also, features such as Quality of Service (QoS) that used to be in the domain of newer technologies such as ATM are incorporated into the Ethernet standard as the technology evolves. The network adapter is the pathway into the server. All requests to the server and all responses from the server must pass through the network adapter. Its performance is key for many server applications. Most LAN adapters are comprised of components that perform functions related to the following:
Network interface/control Protocol control Communication processor PCI bus interface Buffers/storage
The relationship between the network client workstation and the server is called the request/response protocol. Any time a network client wants to access data on the server, it must issue a request. The server then locates the data, creates a data packet, and issues the transmit command to the server LAN adapter to acquire that data from memory and transmit it to the network client workstation (response). This request/response relationship is a fundamental characteristic that also affects server performance.
236
Tuning IBM System x Servers for Performance
Note: In this chapter, we refer to data chunks transmitted over the network by different names, depending on their respective position in the OSI reference model. These are: Ethernet frames (Layer 2) IP datagrams (Layer 3) TCP segments (Layer 4) While the term packet is often used for IP datagrams, we use this term for any data type that resides on Layer 3 or above. The LAN adapter performs two fundamental operations for each packet: The LAN adapter communication processor must execute firmware code necessary to prepare each packet to be moved to or from system memory. This is called adapter command overhead. Every LAN adapter has a limit on the number of packets per second that it can process. Because it takes a certain amount of time to execute this code, the adapter packet throughput rate is reached when the onboard communication processor reaches 100% utilization. In addition to adapter communication processing, each packet that is received or sent by the server requires device driver, TCP/IP, and application processing. Each of these components requires server CPU utilization. Often, one or more of these components can cause packet rate bottlenecks that can result in high-CPU utilization and high interrupt frequency on the server. These bottlenecks can occur when a server is sending or receiving a high percentage of small packets (that is, packets of less than 512 bytes). In this case, LAN utilization might be quite low because the packets are small in size and the amount of data traversing the LAN is small. Note: Never assume that you do not have a LAN bottleneck by simply looking at LAN sustained throughput in bytes per sec. LAN adapter bottlenecks often occur at low LAN utilization but at high sustained packet rates. Observing packets per second often yield clues to these types of bottlenecks. The LAN adapter must also act as a PCI bus master and copy all packets to or from memory. The speed at which this can be sustained determines the adapter DMA throughput capability. When a LAN adapter is moving large amounts of data per packet (approximately 1000 bytes or more), the amount of time spent processing firmware-commands, device driver, and TCP/IP command overhead is small
Chapter 10. Network subsystem
237
compared to the time necessary to copy the large packet to or from server memory. Consequently, it is the DMA performance of the adapter, not the onboard communication processor, that limits packet throughput of the adapter transferring large packets. This is also true for the CPU time of the server processors. With large size packets, the server CPU must spend the majority of time copying data from device driver buffers to TCP/IP buffers and from TCP/IP to the file system buffers. Because of all the server copies, sustained throughput is often determined by the speed of the front-side bus and memory, or simply, the speed at which the server CPUs can move data from one buffer to the next, in other words, how many bytes per second the adapter can move to or from memory.
10.1.1 LAN and TCP/IP performance With the Internet, TCP/IP has replaced most other protocols as the network protocol of choice for networks of all sizes. Windows Server 2003 and Linux use TCP/IP as their default network protocol. TCP/IP enables server software vendors to standardize on one common protocol instead of having to support three or four protocols in their products. While most applications still support other protocols in addition to TCP/IP, support for these protocols is gradually phasing out and features are often only supported with TCP/IP. For example, both Lotus Domino clustering and Microsoft clustering require TCP/IP to function properly. TCP/IP processing accounts for a large amount of overhead in network adapter operations and is often the limiting factor when it comes to network throughput. To understand this process, let us take a look at the actual processing that takes place during TCP/IP operations.
238
Tuning IBM System x Servers for Performance
Figure 10-1 shows the primary components that make up the server. More specifically, it shows the PCI bridges in a dual-peer PCI bridge architecture with a single LAN adapter installed in the right PCI bus segment. Server memory Operating system CPU
CPU
CPU
CPU
Cache
Cache
Cache
Cache
Front-side bus
File system buffers
o
p
Memory controller
TCP/IP process
NIC device driver buffers
PCI slot
PCI bridge
Ethernet
PCI bridge
PCI slot
PCI slot
PCI slot
n
Figure 10-1 Internal data path for TCP/IP
The flow of traffic is as follows: 1. An application that executes in the network client workstation makes a request to the server for data. The networking layer in the client workstation builds a network frame with the address of the target server. The LAN adapter in the client workstation sends the frame as a serial data stream over the network using the address of the target server. 2. The frame arrives at the server LAN adapter as a serial bit stream. It is validated for correctness by the protocol control logic and assembled into a frame in adapter storage. 3. An interrupt or handshake is generated by the LAN adapter to gain service from the server CPU. The server CPU executes the LAN adapter device driver, which responds to the interrupt by building a receive-frame command in a buffer in server memory. The receive-frame command includes a server memory address (destination address for the incoming frame) that tells the LAN adapter where to store the received frame.
Chapter 10. Network subsystem
239
4. The LAN adapter device driver instructs the LAN adapter to gain access to the PCI bus and retrieve the receive-frame command for processing. The LAN adapter gains access to the PCI bus and copies the receive-frame command from server memory to the LAN adapter's onboard buffer. The LAN adapter's communication processor parses the receive-frame command and begins to perform the receive-frame operation. 5. The LAN adapter gains access to the PCI bus (bus master) and uses the destination (server memory) address as the location to store the received frame. The bus master then moves the received frame's contents into the server's receive-buffer memory for processing using direct memory access (DMA). This part of the process is shown by line number one (n) in Figure 10-1 on page 239, which points to the NIC device driver buffer. This step is where the speed of the PCI bus really matters. A higher speed PCI bus enables faster transfers between the LAN adapter and server memory. However, after the data is transferred into the buffer, the PCI bus is no longer used. 6. After the data arrives into the buffer, the adapter generates an interrupt to inform the device driver that it has received packets to process. 7. The IP protocol and TCP process the packet to complete the transport protocol process. 8. After the TCP/IP protocol processing is finished, the NDIS driver sends an interrupt to the application layer that it has data for the server application. The NDIS driver copies the data from the TCP/IP buffer into the file system or application buffers. In Figure 10-1 on page 239, this part of the process is shown by the line pointing into the CPU (o) and the line pointing into the file system buffers (p). Each of these copies is also executed by a server CPU. As shown in Figure 10-1 on page 239, TCP/IP requires up to three transfers for each packet. A server CPU executes all but one of these transfers or copies. Remember that this data must travel over the front-side bus up to two times, and the PCI bus is used for one transfer. A server that is moving 75 MBps over the LAN is doing three times that amount of traffic over the memory bus. This transfer rate is over 225 MBps and does not include the instructions that the CPUs must fetch and the overhead of each packet. Some network adapters that have advanced network features support will greatly improve performance here. For more information, see 10.3, “Advanced network features” on page 259.
240
Tuning IBM System x Servers for Performance
Frames are transmitted to network clients by the server LAN adapter in a similar manner. The process is as follows: 1. When the server operating system has data to transmit to a network client, it builds a transmit command that is used to instruct the LAN adapter to perform a transmit operation. Included inside the transmit command is the address of the transmit frame that has been built in server memory by the network transport layer and initiated by the operating system or server application. The transmit frame includes data that was requested by the network client workstation and placed in server memory by the server operating system or application. 2. After receiving the transmit command, the LAN adapter gains access to the PCI bus and issues the address of the transmit frame to the PCI bus to access server memory and to copy the transmit frame contents into its onboard buffer area. 3. The communication processor on the LAN adapter requests access to the network. 4. When network access is granted, the data frame is transmitted to the client workstation. This explanation is an oversimplification of a very complex process. It is important, however, to gain a high-level understanding of the flow of data from the LAN adapter through the server and the contribution of TCP/IP overhead to server performance. Much of the TCP/IP processing has been omitted, because that processing is complex and beyond the scope of this IBM Redbook. For more detail on Windows Server 2003 TCP/IP operation, see the Microsoft white paper Microsoft Windows Server 2003 TCP/IP Implementation Details, which is available from: http://www.microsoft.com/downloads/details.aspx?FamilyID=06c60bfe-4d374f50-8587-8b68d32fa6ee&displaylang=en The most important point to remember is that data is passed from the LAN adapter to a device driver buffer by LAN adapter bus master transfers. These transfers consume little, if any, server CPU cycles because they are performed entirely by the LAN bus master adapter. After the LAN adapter has moved data into device driver buffers, the transport protocol stack processes the data. After this data is copied from the server transport buffers into the file system memory, the CPU processes the data again. These CPU copies of the data can consume a significant amount of server CPU utilization and front-side bus bandwidth and can create bottlenecks within the server that often limit LAN throughput scalability.
Chapter 10. Network subsystem
241
10.2 Factors affecting network controller performance There are a number of aspects of a server’s configuration that will affect the potential data throughput of a Gigabit Ethernet controller. The factors discussed here are:
10.2.1, “Transfer size” on page 242 10.2.2, “Number of Ethernet ports” on page 246 10.2.3, “CPU and front-side bus” on page 251 10.2.4, “Jumbo frames” on page 255 10.2.5, “10 Gigabit Ethernet adapters” on page 256 10.2.6, “LAN subsystem performance summary” on page 257
10.2.1 Transfer size It is often stated that a Gigabit Ethernet controller can transfer 1000 Mbps (bits) or 100 MBps (bytes). Depending on the type of traffic that needs to be transmitted or received, this transfer rate can actually be more or significantly less, even under ideal conditions. The amount of data that can be transferred over an Ethernet connection depends greatly on the average size of the data packets that need to be transmitted. Because the size of an Ethernet packet is fixed and the CPU and network adapter will have to process each packet regardless of the amount of payload data it carries, small data sizes can overload the CPU subsystem before the maximum theoretical throughput can be reached. By comparison, a full duplex Gigabit Ethernet connection can transmit and receive data at the same time, allowing for transmissions of more than 100 MBps.
242
Tuning IBM System x Servers for Performance
Figure 10-2 shows the data throughput and corresponding CPU utilization for a dual processor server with a single Gigabit Ethernet controller. Throughput and CPU utilization are measured for increasingly larger application I/O transfer sizes. Applications with small transfer sizes include instant messaging and simple file/print while a database and virtualization software are examples of applications with large transfer sizes. Maximum Throughput Comparison at 70% Client Reads / 30% Client Writes 180
50
160
45
p
40
o
120
35 30
100
25 80
20
% CPU Utilization 60
15
Throughput 40
% CPU Utilization
Througput (MBps)
140
Configuration: 2-way Xeon 2.66 GHz 1333 MHz FSB Broadcom 5708 Ethernet Single port
10
20
5
n
0 256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
128
64
0
Application I/O transfer size
Figure 10-2 Network throughput dependency on transfer size
The left Y axis in Figure 10-2 represents throughput in MBps and is shown as the solid line. The throughput plot is the aggregate throughput that is sustained by the server. Throughput continues to increase to about 160 MBps. This increase is because 160 MBps is about the maximum throughput that we can expect for a single Gigabit Ethernet adapter. The right Y axis is the overall CPU utilization. CPU utilization when packet sizes are small is high and can be the source of a bottleneck. CPU utilization for large packet transfers is much lower. To understand LAN throughput behavior, we examine this data more closely , starting with small packet sizes.
Chapter 10. Network subsystem
243
Note: The mix of traffic that passes through the adapter has a big effect on maximum throughput. At a 50-50 ratio of client read-write, throughput would approach 225 MBps—the maximum possible throughput of the adapter—and a 100-0 read-write mix yields a maximum of about 110 MBps. However, a more realistic load of 70-30 client read-write (as shown in Figure 10-2), throughput peaks at approximately 160 MBps.
Small packet sizes In the case of a very small packet, 128 bytes (n in Figure 10-2 on page 243), we see that the server sustains about 18 MBps of total throughput. Any application that sends data in small packets cannot expect to scale because the red dotted line (the right Y axis) shows that two CPUs are at high utilization because it is processing application, TCP/IP, and device driver packet overhead as fast as the small packets can be received and sent. Throughput at 128 byte packets is dominated by adapter and server CPU processing. These components limit throughput to about 18 MBps. One MBps is equal to 1024x1024 bytes or 1,048,576 bytes per second. At 18 MBps, the server is supporting a throughput of 18 x 1,048,576 = 18,874,368 bps. If each packet is 128 bytes, then the server is handling 147,456 packets per second. The worst case scenario is that each packet requires a server interrupt to receive the command and one server interrupt to receive or transmit the data, so this server might be executing over 147,456 x 2 = 294,912 interrupts per second, which is a massive overhead. Most vendors optimize the LAN device driver to process multiple frames per interrupt in order to reduce this overhead. However, in many cases, it is not unreasonable to see servers that execute many thousands of interrupts per second. Assuming that all is operating correctly, when you see this level of interrupt processing, you can be fairly sure that the server is processing many small network packets. Usually, the only solution is to obtain a newer optimized NIC driver capable of servicing multiple packets per interrupt or upgrade to a significantly faster CPU. Care should be exercised when upgrading to a faster CPU because system implementation of memory controller and PCI bus bridge will often limit how fast the CPU can communicate with the PCI LAN adapter. Often this latency can be a significant component of LAN device driver processing time. A faster CPU has little effect on how quickly the CPU can address the PCI LAN adapter on the other side of the memory controller. This bottleneck will usually limit LAN throughput gains when upgrading to faster CPUs.
244
Tuning IBM System x Servers for Performance
Larger packet sizes As we work our way to the right of the chart to study performance for larger packet sizes (p in Figure 10-2 on page 243), notice that throughput for the single LAN adapter increases to a maximum of about 160 MBps. In full-duplex mode, this is about the most we should expect from a single 100 MBps Ethernet NIC. In general, when it come to network performance, the actual amount of data that can be sent across the network depends greatly on the type of application. Take for instance a chat server application that typically sends and receives small messages. Sending a chat message such as “Hey Lou, what are you doing for lunch” is never going to be as large as 8 KB in size. As a result, a chat server application that uses TCP/IP and Ethernet would not scale throughput beyond a single 100 MBps Ethernet NIC because the single NIC would consume all the CPU power most modern servers can provide executing the application, TCP, IP, and device driver overhead. This is not necessarily bad. After all, 8 MBps represents a good many chat messages. However, do not make the mistake of expecting the same performance improvement or scaling for all application environments.
Inefficient frame payloads Do note the dip in throughput at the 1480 byte packet size (o in Figure 10-2 on page 243). This drop in throughput is related to the Ethernet protocol. Standard Ethernet adapters have maximum frame size of 1518 bytes. This is 1500 bytes of maximum transmission unit (MTU) + 14 bytes of Ethernet header + 4 byte Ethernet CRC. About 40 of these bytes must be used for IP and TCP addressing, header, and checksum information. This leaves 1460 bytes for data to be carried by the packet. Note: The original Ethernet standard defined the maximum frame size as 1518 bytes. This was later extended to 1522 bytes to allow for VLANs. Both variants can carry a maximum payload of 1500 bytes. A 1480 byte request overflows the 1460 bytes of data payload of a single packet by 20 bytes. This forces the transport to use two different packets for each 1480 bytes of data being requested; 1460 bytes fill one full packet and the remaining 20 bytes are transmitted in a second packet to complete each 1480 byte request. This of course requires two trips down to the LAN adapter. The overhead of the second 20-byte frame is what causes the drop in throughput because the server CPU must now work twice as hard to send 1480 bytes compared to sending 1460 bytes that fit into a single Ethernet packet. These numbers do translate to a production environment. If you are building a Web server, for example, keeping the size of your images to an integral multiple of 1460 bytes in size can maximize server throughput because each binary
Chapter 10. Network subsystem
245
object could fit in a single or multiple full Ethernet frames. This increases throughput of the server because the Ethernet connection is running at maximum efficiency.
10.2.2 Number of Ethernet ports Adding Ethernet controllers or using more ports in a single controller is an effective way of increasing network throughput. This increase can prove beneficial as long as the bottleneck is the network adapter and not another subsystem. The performance benefit of adding ports depends very much on the packet size being used. For large packets, throughput scales very well, however for small packets, the benefit is less. Figure 10-3 shows this going from one port to two ports doubles the throughput. Maximum throughput comparison at 70% client reads / 30% client writes 350
Server configuration: 2-way Xeon 2.6 GHz 1333 MHz FSB Single/dual BCM 5708 TOE Gigabit Ethernet 70-30 read-write
300 Througput (MBps)
Dual port
250 200 150 Single port
100 50
256k
Application IO Size
64k
16k
4k
1480
1024
256
64
0
Figure 10-3 Throughput comparison of one Ethernet port versus two ports
246
Tuning IBM System x Servers for Performance
Provided the CPU is not a bottleneck (front-side bus is rarely a bottleneck in modern servers), scaling extends beyond adding 4 ports. Figure 10-4 shows an x3755 with four AMD Opteron processors can easily support four Gigabit Ethernet ports. Maximum Throughput Comparison at 70% Client Reads / 30% Client Writes 700
p
Througput (MBps)
600
Server configuration: 4-way 2.6 GHz AMD Opteron Multiport 5706 / 5708 TOE Gigabit Ethernet 70-30 read-write
4 ports
500
o
400
3 ports
300
2 ports
200
n
100
1 port 256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
64
128
0
Application IO Size
Figure 10-4 Throughput effect of adding more Ethernet ports
Chapter 10. Network subsystem
247
As you increase the number of Ethernet connections, CPU utilization also increases proportionally, as shown in Figure 10-5. As we explained in “Small packet sizes” on page 244, CPU utilization is highest with small packet sizes, however in this example CPU utilization is only a bottleneck with four Ethernet ports and small packet sizes. Maximum Throughput Comparison at 70% Client Reads / 30% Client Writes 100
Server configuration: 4-way 2.6 GHz AMD Opteron Multiport 5706 / 5708 TOE Gigabit Ethernet 70-30 read-write
90 CPU Utilization (%)
80
4 ports
70 60
3 ports
50 40
2 ports
30 20
1 port
10
256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
128
64
0
Application IO Size
Figure 10-5 CPU utilization effect of adding more Ethernet ports
Small transfer sizes Examine the chart at the 128 byte transfer size (n in Figure 10-4 on page 247). Regardless of the number of network adapters installed, throughput is always under 45 MBps. For small transfer sizes, the CPU is often the bottleneck. However, notice that in this instance, the CPU is not the bottleneck (Figure 10-5 shows the CPU is not a bottleneck for 1, 2 or 3 ports). In this configuration, the Ethernet adapters have TCP/IP Offload Engine (TOE) capability and are operating in that mode. Here, the limitation is simply the TOE processor on the network card. This is a good thing as far as CPU is concerned—If this configuration did not use TOE, the CPU utilization would most likely be the bottleneck.
248
Tuning IBM System x Servers for Performance
Larger transfer sizes With applications that use large transfer sizes, however, network throughput scales almost linear as additional network cards are installed in the server. Throughput for an application that uses 16 KB packets would scale throughput very nicely up to four Ethernet adapters (see p in Figure 10-4 on page 247). The dip in throughput that we described in “Inefficient frame payloads” on page 245 is more apparent in o in Figure 10-4 on page 247, especially with four network cards.
Linear scaling As mentioned earlier, throughput increases in this configuration are linear across all transfer sizes when no other subsystems are the bottleneck. The same data shown in Figure 10-4 on page 247 plotted differently shows the linear scaling from 1 port through to 4 ports, Figure 10-6. At the smallest transfer sizes four ports gives a 2.5x improvement in throughput, however at the largest transfer sizes, it is a 4x improvement. The scaling is typically less at small transfer sizes either because the TOE processor on the network adapter is the bottleneck (as is the case here) or the server’s CPUs are the bottleneck (as is often the case with non-TOE network controllers and less CPU capacity). Another factor when considering network scaling is the new capabilities of operating systems such as Windows Server 2003 with Receive-side scaling (RSS) as described in 10.3.5, “Receive-side scaling” on page 275.
Chapter 10. Network subsystem
249
Port scaling - Throughput 700 4x increase
Throughput (MBps)
600
32 KB transfers
500 400 300
Increasing transfer sizes 64B - 256KB
Server configuration: 4-way 2.6 GHz AMD Opteron Multiport 5706 / 5708 TOE Gigabit Ethernet 70-30 read-write
200 100 2.5x increase
0 1 port
2 port
3 port
4 port
Figure 10-6 Throughput scales between 2.5x and 4x, going from 1 port to 4 ports
If you compare this four-way x3755 AMD Opteron system with an older 2-way 3.2 GHz Xeon system (compare the red throughput line in Figure 10-7), you can see that scaling is linear up to three Gigabit ports but CPU utilization peaks and limits any further improvement. This shows that the CPU subsystem plays an important part in the performance subsystem.
Gigabit Ethernet scaling 32 KB transfer size 120
Throughput
350
100
CPU
300
80
250
60
200
40
150
20
100
CPU utlization
Throughput (MBps)
400
Server configuration: xSeries 346 Two 3.2 GHz CPUs 800 MHz FSB 2x Intel Pro/1000 MT 2x Broadcom 5704 70-30 read-write Ports configuration: 1. 1x Intel 2. 2x Intel 3. 2x Intel & 1x BCM 4. 2x Intel & 2x BCM
0 1
2
3
4
Number of ports
Figure 10-7 Benefit of adding multiple adapters (70-30 read-write mix)
250
Tuning IBM System x Servers for Performance
10.2.3 CPU and front-side bus Now that we have taken a look at existing LAN subsystem performance, let us turn our attention to the contribution of the server to sustained LAN throughput. As discussed in 10.1.1, “LAN and TCP/IP performance” on page 238, sustained throughput for Ethernet running TCP/IP is largely dependent upon how fast the processors can do buffer copies (bcopies) of data between device driver, TCP/IP, and file system buffers. The bcopy speed is dependent largely upon the speed of the front-side bus and the speed of main memory. The faster the bus and memory, the faster the processors can move the data.
Processor speed When comparing processor impact on performance, and removing the front side bus and adapter as a bottleneck, as a rule of thumb, the expected improvement in performance is half the difference in the speed increase in the processor. The larger the block size, the less the CPU is a bottleneck. So, the benefit of the processor speed is diminished. In Figure 10-8 you see the effect of a higher CPU speed on network throughput. The chart shows a comparison of a 2.66 GHz processor versus a 3.0 GHz processor (a 12% increase) that results in about a 5% increase in throughput at low transfer sizes. A performance benefit is observed at smaller block sizes because the CPU is the bottleneck and the higher clock speed means it is able to process more headers per second. At larger block sizes, performance is almost identical because the CPU is copying data more and is not the bottleneck. Tip: The 2:1 rule of thumb (a 2% increase in CPU speed results in a 1% increase in network throughput) only applies when the front-side bus speed is constant and only for smaller transfer sizes where CPU is the bottleneck.
Chapter 10. Network subsystem
251
Maximum Throughput Comparison at 70% Client Reads / 30% Client Writes 350
15.0% 3.0 GHz Xeon
10.0%
250
2.66 GHz Xeon 200
5.0% 150
Difference 100
Difference
Througput (MBps)
300
Configuration: 1.3 GHz FSB Dual Port 5708 TOE Gigabit Ethernet controller Windows Server 2003 2.66 GHz to 3.0 GHz is a 12% increase.
0.0%
50 0 64k
128k 256k
8k
16k 32k
2k 4k
1480
512
1024 1460
64
128 256
-5.0%
Application IO Size
Figure 10-8 Processor speed scaling (70-30 read-write mix)
Number of processors Adding a second processor will not improve network throughput in most cases, unless the number of network ports is also added. This is because NICs are not able to use multiple processors. A NIC is usually associated to a single CPU that is a bottleneck there processing packet headers as shown in Figure 10-9. Note: Microsoft published a new Scalable Network Pack receive-side scaling (RSS) feature. If this software is installed, a NIC is no longer associated to a single CPU. If the CPU is the bottleneck on a multi-processor server, RSS improves the performance dramatically. For more information, see 10.3.5, “Receive-side scaling” on page 275.
252
Tuning IBM System x Servers for Performance
Maximum Throughput Comparison at 70% Client Reads / 30% Client Writes 100
350
90
300
70
200
60 50
150
40
100
30
CPU Utilization (%)
Througput (MBps)
80 250
Configuration: 1.3 GHz FSB Dual Broadcom 5708 TOE Gigabit Ethernet controllers Windows Server 2003 70-30 read-write
20 50
10
0 256k
64k
128k
32k
8k
16k
4k
2k
1480
1460
1024
512
256
64
128
0
Application I/O Size
1 socket MBps
2 socket MBps
1 socket CPU
2 socket CPU
Figure 10-9 Throughput on a one-CPU and two-CPU system (70-30 read-write mix)
Hyper-Threading Hyper-Threading, a feature of the Pentium 4 family of processors that allows a single physical processor to execute two separate code streams (threads) concurrently. To the operating system, a processor with Hyper-Threading appears as two logical processors, each of which has its own architectural state.
Chapter 10. Network subsystem
253
As long as there is no contention for any of the shared resources this should result in benefits of approximately 15%. Figure 10-10 shows that the network subsystem does gain some benefit from Hyper-Threading at smaller block sizes where CPU is the bottleneck.
Effect of Hyper-Threading Throughput 20%
300 250 200 150
15%
% delta
10%
Hyper-Threading enabled
100 50
5% 0% -5%
Hyper-Threading disabled
Configuration: xSeries 346 Dual 3.2 GHz CPU 800 MHz FSB Dual Broadcom 5721 Windows Server 2003 70-30 read-write
-10% -15%
64 128 256 512 1024 1460 1480 2 KB 4 KB 8 KB 16 KB 32 KB 64 KB 128 KB 256 KB
0
% delta
Throughput (MBps)
350
Transfer size (bytes)
Figure 10-10 The impact of Hyper-Threading on throughput (70-30 read-write mix)
254
Tuning IBM System x Servers for Performance
Figure 10-11 shows the CPU utilization that corresponds to the throughput measurements in Figure 10-10. Notice that CPU utilization is higher when Hyper-Threading is disabled.
Effect of Hyper-Threading CPU utilization Hyper-Threading disabled
CPU utilization (%)
100 90
Hyper-Threading enabled
80 70 60 50 40
Configuration: xSeries 346 Dual 3.2 GHz CPU 800 MHz FSB Dual Broadcom 5721 Windows Server 2003 70-30 read-write
256 KB
64 KB
128 KB
32 KB
8 KB
16 KB
4 KB
1480
2 KB
1460
512
1024
256
64
128
30
Transfer size (bytes)
Figure 10-11 The impact of Hyper-Threading on CPU utilization (70-30 read-write mix)
Ideally, with Hyper-Threading enabled, CPU utilization is halved for the same about of throughput. However, this ideal is only approached as transfer sizes increase. As shown in Figure 10-11, at 256 KB transfers, utilization is about 55% of the non-Hyper-Threading utilization.
10.2.4 Jumbo frames The term jumbo frame refers to an Ethernet frame in excess of the standard 1500-byte size. Bigger packets mean bigger payloads and consequently fewer packet headers per second to process. Using 9000-byte frames can increase network packet throughput, while simultaneously decreasing CPU utilization. Jumbo frames technology is not strictly an alternative to TOE and I/OAT, described in 10.3, “Advanced network features” on page 259, because TOE and I/OAT do not offload processing onto the network controller. Both TOE and I/OAT
Chapter 10. Network subsystem
255
provide throughput equivalent to what jumbo frames offers, and CPU offloading in excess of what jumbo frames offers. However, for those servers that lack TOE or I/OAT controllers, jumbo frames can offer the following benefits: The transmit and receive throughput can improve by up to 5% due to better packing of frames. Jumbo frames can reduce CPU utilization, as compared with standard frames, by transmitting or receiving large chunks of data without requiring segmentation and reassembly up and down the TCP/IP stack. For those customers who have end-to-end support for jumbo frames in their infrastructure, it is available in specific IBM System x servers, including those that use Broadcom BCM5706 or BCM5708 controllers. Note that the requirement for jumbo frames is an end-to-end support for the technology throughout the entire infrastructure. If you want to use jumbo frames, every network controller, switch, storage device, and so on, that is connected through Ethernet needs to have support for jumbo frames. Jumbo frames improve throughput by about 5% at larger blocks due to better packing of frames and decrease CPU utilization by transferring large blocks. The decrease in CPU utilization is the main reason why jumbo frames are deployed.
10.2.5 10 Gigabit Ethernet adapters 10 Gigabit Ethernet adapters are the latest iteration of network adapters that deliver increased network throughput. They follow the IEEE 802.3ae standards and vary slightly to previous Ethernet adapters as they only operate in full duplex mode making collision-detection protocols unnecessary. Initial adapters will also only function over optical fiber. 10 Gigabit Ethernet is capable of operating at in excess of 2.5 GBps during burst mode. This of course, presumes that the other subsystems are capable of supporting that throughput—at this rate the current PCI bus, front-side bus and memory bus, as well current server CPUs would be saturated. Table 10-1 shows the impact of 10 Gigabit Ethernet on the various subsystems. This assumes that the sustained throughput will be 80% of burst mode and full-duplex. When reviewing this table, it is important to remember that the first data transfer to memory is DMA and, therefore, does not have to travel across the front-side bus. There are four remaining data moves across the front-side bus to move the data through to the application.
256
Tuning IBM System x Servers for Performance
There are also two rows in the table for data moving across the front-side bus. This shows the difference between using a TOE-enabled adapter and a non-TOE adapter. The number of data transfers to memory still remains the same, but the data transfers across the front side bus will be reduced. All 10 Gigabit Ethernet adapters should be TOE compliant. Table 10-1 10 Gigabit Ethernet throughputs
Subsystem
Burst Mode
Sustained
PCI adapter
2.24 GBps
1.79 GBps
Front-side bus (without TOE)
8.96 GBps
7.16 GBps
Memory bus
11.2 GBps
8.96 GBps
Front-side bus (with TOE)
4.48 GBps
3.58 GBps
Front-side bus speed minimums
1200 MHz
1200 MHz
10.2.6 LAN subsystem performance summary The key points raised in this section are: A fundamental rule of server performance analysis. A server with a bottleneck will run only as fast as the bottlenecked component will allow, no matter how fast the other parts run. For example, a 64-bit 66 MHz PCI bus can burst 533 MBps. A single-port Gigabit Ethernet adapter at a sustained throughput of 140 MBps to 160 MBps is far from saturating that PCI bus. Even a dual-port Gigabit Ethernet adapter that can sustain 300 MBps to 320 MBps will not saturate the PCI bus. Do not test server throughput by performing a single user file copy. LAN adapters function using a request-response protocol. This means the client makes a request for data from the server and the server responds by sending data. In most cases, applications do not flood the server with requests. They typically wait for a response before sending the next request. Therefore, a single client will almost never load the network or server to its maximum throughput. It takes many clients to show maximum network and server throughput. Therefore, do not run a copy command from your workstation over an Ethernet or any network to the server and expect to see wire speed. Although the PCI, memory and front side bus are capable of supporting the sustained throughput of the Gigabit Ethernet adapter, other bottlenecks might prevent this maximum from being reached.
Chapter 10. Network subsystem
257
Applications that transfer data using small packet sizes will result in low throughput and high CPU overhead. Applications that request small blocks of data require the LAN adapter processor to spend a larger percentage of time executing overhead code while the server processor is executing a high percentage of interrupts. Almost every LAN adapter is unable to sustain wire speed at small packet sizes (less than 512 bytes). Most NIC device drivers do not scale well with SMP. Only one thread can be communicating with an adapter hardware interface at any one time, so having more than two CPUs does not usually produce significant improvements in throughput. Usually, the only solution for increasing performance for small frame environments is to obtain a newer optimized NIC driver capable of servicing multiple packets per interrupt or upgrade to a significantly faster CPU. Care should be exercised when upgrading to a faster CPU, because system implementation of the memory controller and PCI bus bridge will often limit how fast the CPU can communicate with the PCI LAN adapter. Often this latency can be a significant component of LAN device driver processing time. A faster CPU has little effect on how quickly the CPU can address the PCI LAN adapter on the other side of the PCI bus. This bottleneck often limits LAN throughput gains for small packet environments when upgrading to faster CPUs. Solutions exist to help with SMP scalability. See 10.3.5, “Receive-side scaling” on page 275 for information. Transfer size makes a significant difference to throughput. LAN adapters are efficient when applications generate requests for large packets. Ethernet has a payload of 1448 bytes. Planning your objects to be no larger than 1448 bytes or an even multiple of 1448 bytes is best for Ethernet. Windows Server 2003 uses packet segmentation to offload CPU utilization to the Ethernet adapter. This feature offloads the segmentation of large packet requests onto the Ethernet adapter. Basically, this means that rather than have the server CPU segment large transfer requests into suitable size packets, the Ethernet adapter can accept one large transfer request and break the data up into multiple packets. This usually occurs for requests larger than 2 KB in size and explains why CPU utilization begins to decrease after the 2 KB size requests.
258
Tuning IBM System x Servers for Performance
Windows Server 2003 and checksum offload Most Gigabit Ethernet adapters support a Windows Server 2003 function called checksum offload. When packet size exceeds a predetermined threshold, the adapter assumes offload of the checksum function from the server CPU. The checksum is a calculated value that is used to check data integrity. The reason the checksum is offloaded to the adapter for larger packets and not for small packets has to do with performance. At 128-byte packet sizes, a 100 MBps Ethernet adapter might be doing as many as 134,217,728 send and receive packets per second. The processing of checksums for such a high packet rate is a significant load on the LAN adapter processor, so it is better to leave that to the server CPU. As the packet size gets larger, fewer packets per second are being generated (because it takes a longer time to send and receive all that data) and it is prudent to offload the checksum operation on to the adapter. LAN adapters are efficient when network applications requesting data generate requests for large frames. Applications that request small blocks of data require the LAN adapter communication processor to spend a larger percentage of time executing overhead code for every byte of data transmitted. This is why most LAN adapters cannot sustain full wire speed for all frame sizes. In this case, the solutions are new applications (difficult and perhaps impossible) or additional subnetworks using multiple LAN adapters. Faster LAN adapter technology could be used, but the gains would be minimal. Faster LAN technology offers higher data rate, but when the frames are small, a greater percentage of time is spent in adapter overhead and not in data transmission.
10.3 Advanced network features In many cases, it is not the NIC itself but other server components that could be the bottleneck. As a result, new advanced network technologies have been developed that free the CPU and bus subsystems from the heavy workload, thereby increasing performance. This section discusses some of those advanced network features. Note: Technologies such as TOE and I/OAT provide the most benefit at large block sizes. In fact, at small block sizes, there is very little benefit. In addition, the benefit is generally a drop in CPU utilization, while the throughput is unaffected.
Chapter 10. Network subsystem
259
10.3.1 TCP offload engine Processing TCP/IP traffic can consume significant network, memory, CPU and front-side bus resources. As described in 10.1.1, “LAN and TCP/IP performance” on page 238, a TCP/IP request requires multiple trips into and out of the CPU. When processing TCP/IP requests, the CPU is involved in the following activities:
Packet processing Data movement Context switching Interrupt processing
TCP offload engine (TOE) is a hardware-based solution that removes the burden of IP processing from the CPU on a server and moves it down to the NIC. Data is written directly to the NIC, and it handles the IP processing that is necessary to transmit and to receive in the network. TOE frees up CPU for small blocks, but there is still considerable overhead from interrupt processing and context switching on the CPU. For large blocks, the data movement is far more efficient because a single copy is completely eliminated.
260
Tuning IBM System x Servers for Performance
Figure 10-12 compares the dataflow of a traditional Ethernet transfer versus that using a TOE controller.
Standard Ethernet
Ethernet with TOE
CPU
CPU
p q DIMMs
DIMMs
North bridge
North bridge DIMMs
o
South bridge
DIMMs
n
p
q
Gigabit Ethernet adapter
LAN
Data flow: 1. Packet received by NIC and moved to driver/kernel memory space 2. Network controller interrupts CPU to signal arrival of packet 3. CPU processes TCP/IP headers 4. Data is copied from kernel memory space to user memory space by the CPU
South bridge
no Gigabit Ethernet TOE adapter
LAN
Data flow: 1. Packet received by NIC and loaded into TOE engine 2. TOE processes TCP/IP headers 3. Data is copied from NIC memory to user memory space by the TOE engine 4. The NIC interrupts the CPU to indicate the packet is available to the user
Figure 10-12 Comparing standard Ethernet data flow with that of a TOE-enabled system
TOE has two potential benefits: reduced CPU utilization and improved network throughput. As you can see in the right hand side of Figure 10-12, there is very little CPU involvement in the TOE dataflow. The reduction in CPU utilization comes about because it no longer needs to perform the read/modify/write memory sequences that were shown in the standard Ethernet networking model (left side of Figure 10-12). This can be important in a server environment if there are other workloads that are restricted by lack of CPU processing power. TOE can improve network throughput by reducing the interaction required from the
Chapter 10. Network subsystem
261
CPU. The more efficient data movement structure can allow for better flow speeds. If a server has a Gigabit Ethernet adapter operating at a maximum throughput of 220 MBps, it needs the CPU to process approximately 150 000 memory copies per second (assuming 1460-byte packets - 220M/1460) to make the data available to the application without delay, not including the CPU cycles required to process error checking, TCP or checksum validation. This processing places a high load on network resources. Using an adapter that supports TOE can reduce dramatically the impact on the CPU by changing the packet transfer model. TOE will cut copies down to a single DMA operation to user space if the application posts buffers. So TOE will result in 0 copies per second being needed. Using these calculations, the number of I/O operations that the CPU would have to process for the Gigabit Ethernet adapter would be reduced from 613,000 I/Os per second to approximately 306,000 I/Os per second, effectively cutting in half the impact on the CPU to process the data. Figure 10-13 shows the operations that are managed traditionally by the operating system as now managed by the TOE adapter.
Applications
TCP
Operating System
IP
TCP MAC Traditional PHY
Ethernet adapter
TOE-based IP Ethernet MAC adapter
PHY
Figure 10-13 A TCP Offload Engine-enabled network adapter offloading OS functions
This technology decreases the workload that the CPU and front-side bus need to do, thereby enabling the server to dedicate potentially strained resources to other tasks. 10 Gigabit adapters are capable of flooding the CPU, PCI bus, memory,
262
Tuning IBM System x Servers for Performance
and front-side bus and using a TOE adapter in this instance will help to reduce the impact on these subsystems. Figure 10-14 compares the network throughput on a system with TOE enabled versus the same system with TOE disabled. The chart shows there are gains at low-medium transfer sizes (as much as 30% at 1024 bytes), but at large transfer sizes, TOE does not provide much gain in throughput.
350
35.0%
300
30.0% 25.0%
250
15.0% 150 10.0% 100
% Delta
20.0% 200
5.0%
256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
-5.0% 512
0 256
0.0%
128
50
64
Througput (MBps)
Maximum Throughput Comparison at 70% Client Reads / 30% Client Writes
Application IO Size 5708 TOE on RSS on 5708 TOE off RSS on %Delta: 5708 TOE on RSS on to 5708 TOE off RSS on
System: BladeCenter HS21; 2-way 2.66 GHz Xeon; Broadcom 5708 dual-port controller; Windows Server 2003 SP1 with SNP; RSS on Figure 10-14 Throughput comparison—TOE enabled versus disabled
Chapter 10. Network subsystem
263
However, comparing CPU utilization in Figure 10-15, you can see that TOE lowers the demand on CPU capacity. The chart shows that there is a drop in CPU utilization at high transfer sizes.
CPU Utilization at 70% Client Reads / 30% Client Writes 20.0%
120
10.0% 100 0.0%
% CPU
-20.0% 60 -30.0% -40.0%
40
-50.0% 20 -60.0% 256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
512
1024
256
128
-70.0% 64
0
Application IO Size 5708 TOE on RSS on 5708 TOE off RSS on %Delta: 5708 TOE on RSS on to 5708 TOE off RSS on
System: BladeCenter HS21; 2-way 2.66 GHz Xeon; Broadcom 5708 dual-port controller; Windows Server 2003 SP1 with SNP; RSS on Figure 10-15 CPU utilization comparison—TOE enabled versus disabled
264
Tuning IBM System x Servers for Performance
% Delta
-10.0%
80
A better way of showing the effect on CPU utilization is to plot the CPU Efficiency which is Throughput/CPU Utilization, where the higher the number the better. This is shown in Figure 10-16. CPU efficiency is equal at low transfer sizes (comparing TOE enabled versus TOE disabled), while CPU efficiency is markedly higher at large transfer sizes when TOE is enabled.
60%
10.0
40%
5.0
20%
0.0
0%
% Delta
15.0
256k
80%
128k
20.0
64k
100%
32k
25.0
16k
120%
8k
30.0
4k
140%
2k
35.0
1480
160%
1460
40.0
1024
180%
512
45.0
256
200%
128
50.0
64
Efficiency (MBps / %CPU)
CPU Efficiency Comparison at 70% Client Reads / 30% Client Writes
Application IO Size 5708 TOE on RSS on 5708 TOE off RSS on %Delta: 5708 TOE on RSS on to 5708 TOE off RSS on
System: BladeCenter HS21; 2-way 2.66 GHz Xeon; Broadcom 5708 dual-port controller; Windows Server 2003 SP1 with SNP; RSS on Figure 10-16 CPU efficiency comparison (70/30% client read/write)
The packet processing capability of a TOE engine is definitely less than a pair of Xeon processors. At small blocks, the number of packets per second is the highest, so TOE will generate a lower throughput level than the host CPUs. So, the host CPU essentially consumes proportionately higher CPU for the higher bandwidth that it delivers. Also, with small blocks, there is no data copying done, only protocol offload. As the block size increases, the TOE adapter will be able to offload data movement and protocol processing, so it can actually generate lower CPU utilization.
Chapter 10. Network subsystem
265
The above charts are for a client read/write ratio of 70/30, common for most production workloads. TOE is most effective, however, when the client write are at 100%, as shown in Figure 10-17.
60.0
500%
50.0
400%
40.0
300%
30.0
200%
20.0
100%
10.0
0%
256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
128
-100% 64
0.0
% Delta
Efficiency (MBps / %CPU)
CPU Efficiency Comparison at 0% Client Reads / 100% Client Writes
Application IO Size 5708 TOE on RSS on 5708 TOE off RSS on %Delta: 5708 TOE on RSS on to 5708 TOE off RSS on
System: BladeCenter HS21; 2-way 2.66 GHz Xeon; Broadcom 5708 dual-port controller; Windows Server 2003 SP1 with SNP; RSS on Figure 10-17 CPU efficiency comparison (100% client writes)
Because the data movement steps are modified (compare the two data flows in Figure 10-12 on page 261), and occur in a different sequence, TOE requires support by driver and operating system. Currently, Windows Server 2003 x64 offers support but Linux does not. TOE is supported by many Ethernet controllers integrated into System x servers, including those with the Broadcom 5708 Ethernet chipset. TOE is also a feature of a number of Gigabit Ethernet adapters including the NetXtreme II 1000 Express Ethernet adapter.
266
Tuning IBM System x Servers for Performance
10.3.2 I/O Accelerator Technology I/O Acceleration Technology (I/OAT or IOAT) is an Intel technology that was developed as an alternative to TOE. Similar to TOE, it reduces performance bottlenecks by offloading work from the processor in several ways. I/OAT is also known as NetDMA. I/OAT is implemented as a chipset in the server. It implements enhanced data moving capabilities, a new API to control data flow, and a new driver. I/OAT is supported by both Windows and Linux. I/OAT provides the following: An optimized protocol stack to significantly reduce protocol processing cycles Header splitting which allows processing of packet headers and payloads on parallel paths. Performs interrupt modulation, to prevent excessive interrupts Uses direct memory access (DMA), to reduce the latency (number of CPU cycles consumed) while waiting for a memory access to finish Even though TCP/IP has been upgraded and enhanced, the core of the TCP/IP protocol stack has remained unchanged since 1977. To drive improvements in TCP/IP performance to keep up with advances in CPU, memory, and PCI technology, Intel has implemented several enhancements to streamline the network stack: Separate data and control paths. These paths are enabled by header-splitting in the network adapter’s media access controller (MAC). Cache-aware data structures. These increase the percentage of cache hits and reduce the number of memory accesses required to process a packet. Improved exception testing. This reduces the length of the path the packet must travel through the protocol stack. Header splitting reduces latency in packet processing by eliminating the time wasted by the CPU looking at both the header and payload as a single entity. The CPU needs to look only at the header to start the delivery process. Other system components can handle the packet payload more efficiently. Header splitting also helps in improving the cacheability of headers—because the headers in a connection are positioned in consecutive addresses, the headers are moved into the cache for processing through processor prefetches. Interrupt modulation allows the CPU to spend more time on other tasks, rather than having to acknowledge each packet.
Chapter 10. Network subsystem
267
Direct memory access (DMA) bypasses the latency that is caused by data movement between memory buffers. Using DMA, as soon as the CPU sends one request off, it can move on to start processing another task. It no longer needs to be involved directly in the movement of data through the processors. The I/OAT work flow is shown in Figure 10-18.
CPU
Data flow: n A packet is received by NIC and direct memory accessed (DMA) to driver/kernel memory space.
CPU
p
o The network controller interrupts the CPU to signal the arrival of the packet.
q DIMMs
North bridge with I/OAT
DIMMs
n
o South bridge
I/OAT Gigabit Ethernet adapter
p The CPU processes the TCP/IP headers. q The data is copied from the kernel memory space to the user memory space by the DMA engine.
LAN
Figure 10-18 Data flow for data received
I/OAT is supported in System x servers with the use of the Intel PRO/1000 PT Dual Port Server Adapter. I/OAT also requires an operating system level update, as described in 10.3.4, “TCP Chimney Offload” on page 273. IOAT is most beneficial when most I/O traffic is written from the client to the server, because the DMA engine is useful only for this workload and RSS helps only this workload. With I/OAT, there is no optimization for client reads. The following three charts compare a system with an I/OAT enabled adapter (the Intel PRO/1000 PT Dual Port Server Adapter) with standard Gigabit adapter (the Broadcom 5708 without TOE enabled)
268
Tuning IBM System x Servers for Performance
Figure 10-21 shows that I/OAT results in lower CPU utilization for large block sizes—as much as half for large block sizes.
CPU Utilization at 0% Client Reads / 100% Client Writes 120
10.0% 0.0%
100
-10.0% -20.0% 60 -30.0%
% Delta
% CPU
80
40 -40.0% 20
-50.0%
256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
128
-60.0% 64
0
Application IO Size IOAT DMA on RSS on 5708 TOE off RSS off %Delta: IOAT DMA on RSS on to 5708 TOE off RSS off
System: I/OAT: System x3650, 2-way 2.66 GHz Xeon, Intel PRO/1000 PT Dual Port Server Adapter; Broadcom 5708 system: BladeCenter HS21, 2-way 2.66 GHz Xeon; BCM5708 dual-port controller; Both systems running Windows Server 2003 SP1 with SNP Figure 10-19 CPU utilization comparison—IOAT enabled versus disabled
Chapter 10. Network subsystem
269
I/OAT results in higher CPU efficiency compared to a non-I/OAT system, as shown in Figure 10-20.
CPU Efficiency Comparison at 0% Client Reads / 100% Client Writes 120%
20.0
100%
16.0 14.0
80%
12.0 60%
10.0 8.0
40%
6.0 4.0
20%
2.0 256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
128
0% 64
0.0
Application IO Size IOAT DMA on RSS on 5708 TOE off RSS off %Delta: IOAT DMA on RSS on to 5708 TOE off RSS off
Figure 10-20 CPU efficiency comparison—IOAT enabled versus disabled
270
Tuning IBM System x Servers for Performance
% Delta
Efficiency (MBps / %CPU)
18.0
As we have discussed, throughput does not benefit from technologies such as I/OAT, as shown in Figure 10-21.
Maximum Throughput Comparison at 0% Client Reads / 100% Client Writes 45.0%
250
40.0% 35.0% 30.0% 25.0%
150
20.0% 15.0%
100
% Delta
Througput (MBps)
200
10.0% 50
5.0% 0.0% 256k
64k
128k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
128
-5.0% 64
0
Application IO Size IOAT DMA on RSS on 5708 TOE off RSS off %Delta: IOAT DMA on RSS on to 5708 TOE off RSS off
Figure 10-21 Throughput comparison—IOAT enabled versus disabled
10.3.3 Comparing TOE and I/OAT TOE and I/OAT are competing technologies that aim to accomplish the same goal: improving system performance by offloading communication workload from the system processor to other components. However, the way that they achieve this goal is different. Each technology has advantages and disadvantages when compared with the other. Either technology might be acceptable for you, depending on your needs and infrastructure; however, TOE and I/OAT do differ in several respects: TOE accelerates both transmit and receive traffic. I/OAT accelerates only receive traffic. TOE offloads data movement and protocol processing. I/OAT offloads only data movement. I/OAT is supported by Linux. TOE currently is not.
Chapter 10. Network subsystem
271
I/OAT is a stateless offload, it does not require state to be stored in the offload engine that scales the number of connections that are offloaded. TOE is not. Both TOE and IOAT require the Linux kernel to be modified. In the case of IOAT, the Linux stack is modified so that during server receives, it instructs the DMA engine in the memory controller to copy the data from the kernel buffers to the user buffers. This is a relatively minor change to the Linux kernel. However, for TOE, the Linux stack has to be rearchitected because essentially TOE is a TCP/IP stack running in parallel to the Linux TCP/IP stack. The stack has to be rearchitected to support NICs that use the Linux TCP/IP stack, TOE adapters that use portions of the kernel stack and offload portions of it, and full offload adapters that complete bypass the standard Linux TCP/IP stack. This would be a major change to the Linux stack and has not yet been implemented. Many customers have already standardized on Ethernet products from either Intel or Broadcom. Ultimately, this standardization might have more bearing on whether you choose TOE or I/OAT than the relative merits of one technology over the other. Both offload technologies offer the biggest benefit with the server is doing large-block receives (that is, 100% client write at large transfer sizes). TOE offloads data movement and protocol processing, while I/OAT offloads data movement only
272
Tuning IBM System x Servers for Performance
Figure 10-22 provides a comparison of I/OAT and TOE showing that CPU utilization is lowered more with TOE than with I/OAT at large block sizes.
120
250.0%
100
200.0%
80
150.0%
60
100.0%
40
50.0%
20
0.0%
256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
128
-50.0% 64
0
% Delta
% CPU
CPU Utilization at 0% Client Reads / 100% Client Writes
Application IO Size IOAT DMA on RSS on 5708 TOE on RSS on %Delta: IOAT DMA on RSS on to 5708 TOE on RSS on
System: I/OAT: System x3650, 2-way 2.66 GHz Xeon, Intel PRO/1000 PT Dual Port Server Adapter; TOE: BladeCenter HS21, 2-way 2.66 GHz Xeon; BCM5708 dual-port controller; Both systems running Windows Server 2003 SP1 with SNP.
Figure 10-22 Throughput comparison—IOAT versus TOE
10.3.4 TCP Chimney Offload TCP Chimney Offload is a Microsoft technology that optimizes server performance when processing network traffic. It is implemented in a Microsoft component, Scalable Networking Pack (SNP). This feature combined with TOE network adapters removes existing operating system bottlenecks such as CPU processing overhead related to use network packet processing and the ability to use multiple processors for incoming network traffic. Applications that are currently bound with network processing overhead will generally scale better when used with TCP Chimney.
Chapter 10. Network subsystem
273
The Scalable Networking Pack is available for the following operating systems: Windows Server 2003 SP1 x32 and Windows Server 2003 R2 x32 Windows Server 2003 SP1 x64 and Windows Server 2003 R2 x64 Windows XP x64 You can download the Scalable Networking Pack from: http://support.microsoft.com/?kbid=912222 The product will be introduced in Windows Vista™ and included in future versions of Microsoft operating systems. It includes network acceleration technology as well as support for hardware offloading based on the TCP/IP protocol supporting IPv4 and IPv6 shown in Figure 10-23.
Transport/TCP (Layer 4)
MS Windows Server 2003 Scalable Networking Pack
OS
Data link/ Media Access Control (Layer 2) Physical (Layer 1) Normal devices
Traffic/data path
Network/IP (Layer 3)
Traffic/data path
Hardware
Software
Application
Network hardware
Transport/TCP (Layer 4) Network/IP (Layer 3) Data link/ Media Access Control (Layer 2) Physical (Layer 1) TOE devices
Figure 10-23 Microsoft Scalable Network Pack implementation
274
Tuning IBM System x Servers for Performance
Figure 10-23 shows that the Scalable Network Pack is embedded in the first two software layers of the OSI reference model. TCP Chimney Offload creates a software switch shown in Figure 10-24, between the top of the protocol stack and the software drivers. Incoming data is transferred directly to the top of the protocol stack, without moving through the intermediate protocol layers. That is also the reason why this technology is called chimney—it ascends like smoke in a chimney. The key concept of the chimney is that data transfer only occurs through the top or the bottom. At the top of the chimney is the implemented switch, which is managed by the operating system. The data that is coming in at the physical layer is getting transferred directly through the chimney to the switch. There, Windows decides either to offload the data back through the chimney to the TOE engine or to process the data itself. If it offloads the data to the TOE engine, it increases the host performance as described in 10.3.1, “TCP offload engine” on page 260.
Application
Switch
Top protocol of chimney Intermediate protocol layers State update interface Intermediate protocol layers Intermediate protocol layers
TOE NIC
Data transfer interface Chimney
Offload target
Figure 10-24 Chimney Offload block diagram
10.3.5 Receive-side scaling As discussed in 10.2.3, “CPU and front-side bus” on page 251, adding a second CPU to a server does not increase networking performance even if the CPU is the bottleneck, because a network adapter in a multi-core server running Windows is associated with a single core. This limitation means that the associated CPU must handle all the traffic, regardless of there are other CPUs available. If there is so much incoming traffic that the TOE and the associated
Chapter 10. Network subsystem
275
CPU are not able to handle all traffic fast enough, the network adapter discards the traffic, resulting in retransmissions and decreased performance. Receive-side scaling (RSS) attempts to solve this problem. RSS is a new Network Driver Interface Specification (NDIS) 6.0 technology. It is primarily a software enhancement that takes advantage of multi-core platforms by distributing network processing across the cores. RSS enables packet receive-processing to scale according to the number of available computer processors. RSS is only available for Windows and is implemented in the Scalable Networking Pack. RSS offers the following benefits: Parallel execution Receive packets from a single network adapter can be processed concurrently on multiple CPUs, while preserving in-order delivery. Dynamic load balancing As the system load on the host varies, RSS rebalances the network processing load between the processors. Cache locality Because packets from a single connection are always mapped to a specific processor, the state for a particular connection never has to move from one processor’s cache to another, thus minimizing cache thrashing and promoting improved performance. Send-side scaling TCP is often limited in how much data it can send to the remote peer. When an application tries to send a buffer larger than the size of the advertised receive window, TCP sends part of the data and then waits for an acknowledgment before sending the balance of the data. When the TCP acknowledgement arrives, additional data is sent in the context of the deferred procedure call in which the acknowledgment is indicated. Thus, scaled receive processing can also result in scaled transmit processing. Secure hash The default generated RSS signature is cryptographically secure, making it much more difficult for malicious remote hosts to force the system into an unbalanced state. To optimize the performance of this parallel processing of received packets it is critical to preserve in-order delivery. If packets are distributed among the cores of a server, and packets of one connection are processed on different CPUs, it is
276
Tuning IBM System x Servers for Performance
not possible to enforce that older packets get processed first, and performance would decrease because that. RSS assures an in-order packet delivery by ensuring that only one CPU processes packets for a single TCP/IP connection. This means a single TCP/IP connection will always be handled by the same CPU, but different TCP/IP connections can be handled parallel on other CPUs. RSS requires that the network adapter checks each packet header to create an hash result that is used as an index into the indirection table, and then added to a base CPU number to determine the processor that the packet should be processed on. Because the host protocol stack can change the contents of the table at any time, the TCP/IP stack can dynamically balance the processing load on each CPU. RSS is most beneficial when CPU is the bottleneck and there are addition CPUs that are currently not being used for I/O processing, as shown in Figure 10-25. Maximum Throughput Comparison at 0% Client Reads / 100% Client Writes 250
200.0% 180.0% 160.0% 140.0% 120.0%
150
100.0% 80.0% 100
60.0%
% Delta
Througput (MBps)
200
40.0% 50
20.0% 0.0% 256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
128
-20.0% 64
0
Application IO Size 5708 TOE off RSS on 5708 TOE off RSS off %Delta: 5708 TOE off RSS on to 5708 TOE off RSS off
System: BladeCenter HS21; 2-way 2.66 GHz Xeon; Broadcom 5708 dual-port controller; Windows Server 2003 SP1 with SNP Figure 10-25 RSS benefits most at small block sizes (where CPU is the bottleneck)
Chapter 10. Network subsystem
277
As shown in Figure 10-26, with RSS on, more CPU power can now be used to process I/O requests.
CPU Utilization at 0% Client Reads / 100% Client Writes 100.0%
120
80.0%
100
60.0% 40.0% 60 20.0%
% Delta
% CPU
80
40 0.0% 20
-20.0%
256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
128
-40.0% 64
0
Application IO Size 5708 TOE off RSS on 5708 TOE off RSS off %Delta: 5708 TOE off RSS on to 5708 TOE off RSS off
System: BladeCenter HS21; 2-way 2.66 GHz Xeon; Broadcom 5708 dual-port controller; Windows Server 2003 SP1 with SNP Figure 10-26 RSS benefits most at small block sizes (where CPU is the bottleneck)
278
Tuning IBM System x Servers for Performance
The result is a more efficient use of the CPU at small transfers because of RSS, as shown in Figure 10-27.
CPU Efficiency Comparison at 0% Client Reads / 100% Client Writes 70% 60%
10.0
50% 8.0 40% 6.0 30%
% Delta
Efficiency (MBps / %CPU)
12.0
4.0 20% 2.0
10%
256k
128k
64k
32k
16k
8k
4k
2k
1480
1460
1024
512
256
128
0% 64
0.0
Application IO Size 5708 TOE off RSS on 5708 TOE off RSS off %Delta: 5708 TOE off RSS on to 5708 TOE off RSS off
System: BladeCenter HS21; 2-way 2.66 GHz Xeon; Broadcom 5708 dual-port controller; Windows Server 2003 SP1 with SNP Figure 10-27 RSS benefits most at small block sizes (where CPU is the bottleneck)
For more information about the Receive-side Scaling, read the Microsoft White Paper Scalable Networking with RSS, which is available from: http://www.microsoft.com/whdc/device/network/NDIS_RSS.mspx Note: Chimney Offload and RSS are both included in the Microsoft Scalable Networking Pack, but they are two independent features. The use of RSS does not require the use off Chimney Offload.
Chapter 10. Network subsystem
279
10.3.6 RDMA overview Remote DMA (RDMA) uses a combination of hardware and software to increase the efficiency of data movement over network connections. The concept is that the reduction in the need for intermediate memory buffering of network data will result in the direct movement of network data between application layers, and therefore improved performance. Direct memory access (DMA) usually uses a hardware device other than the main CPU to move blocks of data between various components within a system. DMA transfers must be configured by system software, but the details of the actual data transfers are left to the hardware. RDMA is an extension of this concept, allowing data to be moved between two networked systems without direct CPU intervention.
280
Tuning IBM System x Servers for Performance
This technology is more complex in that it requires coordination between both ends (platforms) of a network data transfer. Figure 10-28 illustrates the high-level operation of RDMA.
Source system
Target system
CPU
CPU
n, q
n, q DIMMs
DIMMs
North bridge DIMMs
North bridge
DIMMs
p
South bridge
p
LAN connection Gigabit Ethernet
o TOE adapter
Data flow on the source system: n Transfer Requested and Initialized from Source. o Source initiates TOE engine pointers for transfer. p Data is copied from source memory to source NIC by the TOE engine, and moved to LAN. q Transfer acknowledged by source.
Gigabit Ethernet TOE adapter o
South bridge
Data flow on the target system
n Transfer acknowledged and initialized to target. o Target initializes TOE engine pointers for transfer. p Data is received from LAN by target NIC and moved to target memory by the TOE engine. q Transfer acknowledged by target.
Figure 10-28 RDMA data flow
RDMA is a true end-to-end solution, where the application layer of one server talks directly to the application layer of the other server, and this can yield significant performance enhancements under specific situations. However, RDMA also has a major restriction in that both the source and target must cooperate and synchronize pointers in order to complete a transfer. As such, both source and target platforms must fully support RDMA. A key component of RDMA is the level of memory protection needed, because traditional operating system memory boundaries between user memory spaces must be maintained. This requires that both platforms run an RDMA-enhanced
Chapter 10. Network subsystem
281
operating system. At the time of writing, only Microsoft Windows supports RDMA and that support is through the Scalable Networking Pack. You can download the Scalable Networking Pack from: http://support.microsoft.com/?kbid=912222 Note, however, that there are currently no adapters available to take advantage of RDMA. RDMA performance gains can be significant, because applications can transfer data directly to the NIC, bypassing the operating system kernel. The TOE processor offloads the TCP/IP processing as described previously. The result of this is both reduced load on the CPU and reduced latency for data movement. The greatest advantage can be seen in high-performance computing and clustered database applications. However, at the time of writing, RDMA is an evolving technology and not currently in wide use.
10.3.7 Operating system considerations To benefit from these new networking technologies, the operating system view to the network interface must be modified. Microsoft is implementing support for these technologies using its Chimney Offload technology. This software architecture provides the flexibility to offload parts of the network software stack to the network device. Chimney technology, which includes support for TOE, is delivered as part of the Scalable Networking Pack upgrade and is described in 10.3.4, “TCP Chimney Offload” on page 273. Microsoft is also considering the addition of RDMA support to the Scalable Networking Pack, but at the time of writing, these plans had not been set. RDMA is discussed in 10.3.6, “RDMA overview” on page 280. The Linux community currently has no concrete plans to support TOE. However, RDMA, described in 10.3.6, “RDMA overview” on page 280 is
282
Tuning IBM System x Servers for Performance
receiving wide focus and is likely to roll out in Linux first, potentially in clustering applications. Table 10-2 lists the existing support (or expected support, at the time of writing) for the advanced networking features that are described in this chapter. Table 10-2 Expected operating system support for advanced networking features
Operating System
TOE
I/OAT
Jumbo Frames
RSS
RDMA
Linux (Red Hat, SUSE)
No
Yes
OS support not needed
Yes
Future
Windows 2000 Server
No
No
OS support not needed
No
No
Windows Server 2003
Yes, through SNP
Yes, through SNP
OS support not needed
Yes, through SNP
Future
Windows “Longhorn”
Yes
Yes
OS support not needed
Yes
Yes
10.4 Internet SCSI (iSCSI) iSCSI is a transport protocol that carries SCSI commands from an initiator to a target. It is a data storage networking protocol that transports SCSI block I/O protocol requests (commands, sequences, and attributes) over TCP/IP. SCSI data and commands are encapsulated in TCP/IP packets which means that iSCSI enables Ethernet-based Storage Area Networks (SANs) as opposed to Fibre Channel-based SANs. iSCSI is well suited to run over almost any physical network. By eliminating the need for a second network technology just for storage, it can lower the cost of implementing network storage and it offers the capability to extend beyond the confines of a LAN, to include Metropolitan and Wide Area Networks (MANs and WANs). The iSCSI technology is a native IP interconnect that wraps SCSI data and commands in TCP/IP packets. The receiving device takes the command out of the IP packet and passes it to the SCSI controller, which forwards the request to the storage device. When the data is retrieved, it is again wrapped in an IP packet and returned to the requesting device.
Chapter 10. Network subsystem
283
Do not confuse these IP-based SANs with network attached storage (NAS). NAS permits users to use a LAN file system, where for example the access authorization gets managed by the NAS-box itself, it cannot be formatted or have a file system loaded on it. iSCSI on the other hand, delivers a block-based file system. The operating system running on the connected server treats the iSCSI device like an directly in the server attached hard disk and is able to format the disk, has control of the access rights and all so on. You are also able to boot from the attached iSCSI device. For example, an IBM TotalStorage DS300 is an iSCSI target server. For more information about the iSCSI performance see 9.4.3, “iSCSI” on page 189. With iSCSI technology, you can create a SAN from existing, familiar and inexpensive Ethernet components, and to quickly develop SAN skills without a lot of retraining. It affords administrators the ability to centrally control storage devices, to pool storage resources, to integrate NAS appliances into the SAN, and to apply familiar IP security methods to the SAN. An iSCSI SAN not only offers the same kinds of remote backup, clustering, mirroring, disaster recovery and business continuance capabilities as an FC SAN, it goes FC one better by making the offsite distance essentially unlimited. Finally, a SAN can improve the performance of not only the storage and retrieval of data, but of the user IP network as well.
10.4.1 iSCSI Initiators The device at the server end of an iSCSI connection is called an iSCSI initiator and can be either hardware or software based. The iSCSI initiator is responsible for initiating the SCSI request over IP to a target server. Every host that requires access to the iSCSI target must have at least one initiator installed.
284
Tuning IBM System x Servers for Performance
As shown in Figure 10-29, the firmware running on the target device manages the SCSI over IP requests. The initiator intercepts disk access requests and sends commands to the target. Later when the target responds with disk information, the initiator receives the responses and passes them back to the requestor. iSCSI connections use layer 5 (session layer) of the OSI seven layer reference model.
Applications
iSCSI driver TCP/IP Network driver
NIC
Figure 10-29 iSCSI initiators in the OSI layer 7 reference model
Chapter 10. Network subsystem
285
Figure 10-30 shows the encapsulation of iSCSI in TCP/IP packets. The upper packet shows the composition of a normal TCP/IP packet, as used for normal data transfers over TCP/IP Ethernet connections. The packet below shows how the iSCSI initiator and the SCSI data are encapsulated in the TCP/IP packet.
Standard TCP/IP packet
Ethernet Header
iSCSI packet
Ethernet Header
IP
IP
TCP
TCP
SCSI
iSCSI initiator data
Data
CRC
SCSI
Data
CRC
SCSI data
Figure 10-30 iSCSI data packet encapsulation
Software initiators Microsoft and Linux operating systems have iSCSI software initiators. An iSCSI software initiator is an iSCSI driver that works with the TCP/IP stack, network drivers, and NICs to provide iSCSI connectivity to other iSCSi devices through the IP network. Because an iSCSI software initiator depends on the operating system’s IP stack, if the IP stack fails, access to the remote disk is lost. As a result, software initiators are not ideal for booting the server. An iSCSI software initiator is an inexpensive alternative to an iSCSI hardware initiator. It is able to provide iSCSI support through software instead of a separate iSCSi adapter. However, the transfer of the iSCSI IP packets impacts server performance as shown in 10.1.1, “LAN and TCP/IP performance” on page 238. It adds additional workload to the server processor, TCP/IP stack and network driver to handle the additional SCSI protocol and data traffic. Because the host CPU is responsible for processing TCP/IP requests, iSCSI can suffer performance degradation, especially in high traffic settings. This performance limitation is especially significant when compared with Fibre Channel, which does not have TCP/IP overhead. However, iSCSI software initiators are a potential fit for casual demands for storage access.
286
Tuning IBM System x Servers for Performance
One method to address the performance problem is to increase the speed of your host processor. Another method is to use an Ethernet controller with TCP/IP Offload Engine (TOE).
TOE in combination with iSCSI We describe TOE in 10.3.1, “TCP offload engine” on page 260. Using TOE in combination with iSCSI can improve your server performance. TOE offloads, as shown in the IP n and TCP o processing from the CPU and improves performance. However, the iSCSI software initiator p is still required: iSCSI connections use layer 5, the TOE handles processing up to layer 4, and then transmits the packets to the application, as shown in Figure 10-31. TOE handles the IP processing, but the iSCSI initiator is still handled by the host system.
Applications
TCP
o
IP
n
Operating System
TCP MAC Traditional PHY
Ethernet adapter
p
iSCSI
TOE-based IP Ethernet MAC adapter
o n
PHY
Figure 10-31 TOE in combination with iSCSI
Note: If you are using iSCSI in combination with TOE, it might not allow you to run your standard TCP/IP connections off the NIC unless the vendor has provided some type of filter driver to intercept the standard TCP/IP requests.
Hardware initiator The iSCSI initiator or driver can be implemented in a hardware adapter card, rather than in the host software. You can implement the adapter card using an
Chapter 10. Network subsystem
287
iSCSI host bus adapter (HBA). The iSCSI processing is offloaded to the hardware adapter card instead of processing the iSCSI protocol in the host software. iSCSI TCP/IP processing is also offloaded to the TOE engine on the HBA. With both TCP and iSCSI processing on the adapter card, high-speed transport of block data with minimal CPU overhead is possible. The hardware initiator is the more expensive of the iSCSI initiator options as it require the purchase of an iSCSI HBA, but it is the most capable and the best performer. All of the SCSI block-processing and TOE functions are integrated into the HBA. This frees the host CPU from having to do any of the iSCSI processing.
10.4.2 iSCSI network infrastructure In theory, an iSCSI SAN does not need a separate network infrastructure. It is able to run over the existing network switches similar to the normal network traffic. However, in the case of high throughput demand, iSCSI produces a high network load and other protocols and applications will suffer because of that load. Thus, for the iSCSI network traffic, it is advisable to use a separate switch or a separate VLAN and separate NICs to raise the transfer rate. This way, no other protocols or applications will suffer a decrease in performance, and you will have higher security of your delivered iSCSI data. If you are not able to use a separate LAN or if your iSCSI packets have to be securely delivered over the Internet, you might need to use IPsec (IP Security). IPsec is a set of cryptographic protocols for securing packet flows and key exchange. Although Microsoft supports IPsec, at the time of the publication of this book, the big hardware vendors do not currently support IPsec in combination with iSCSI. In the near future, it is expected that there will be iSCSI HBAs available that will be able to offload the IPsec encryption and decryption from the host CPU and increase server performance. Be aware that because accessing iSCSI-attached storage requires traversing a network, there is additional latency when compared to a SCSI or SAS solution. In addition, Ultra320 SCSI (320 MBps peak) is faster than a Gigabit Ethernet connection (a peak of 112 MBps). Another use of iSCSI is remote boot capability. Because SCSI traditionally is used for operating system boot as well as data storage, iSCSI can be used for boot support. At a high level, this is similar to existing network boot methods such as PXE (Preboot Execution Environment) and BootP. IBM support for iSCSI remote boot is limited currently to blade servers and optional iSCSI adapters.
288
Tuning IBM System x Servers for Performance
10.5 Interconnects Today’s networking technology offers high network performance that is adequate for the most server types. However there are some uses like high-performance computing (HPC) that require much more throughput or lower latency. In this section we will discuss two of the interconnect technologies that are the most common in use.
10.5.1 Myrinet Myrinet is a high speed LAN technology. It has much less protocol overhead than Ethernet and is designed to be used as interconnect between the nodes of computer clusters. The nodes are connected through low overhead routers and switches that are connected to the host with a single connector. Also, physical connections are used—two fibre optic cables, one for downstream and one for upstream. Myrinet’s throughput is close to the theoretical maximum of the physical layer. The main benefits of Myrinet are: Better throughput Less interference Less latency while using the host CPU Flow control Error control Switched network design (multiple paths, low latency cut through switches, with monitoring for high availability applications) Heartbeat monitoring on every link
Myrinet is available in two versions Myrinet-2000 (superior alternative to Gigabit Ethernet)—Full duplex data rate: 2 + 2 Gbps Myri-10GB—Full duplex data rate: 10 + 10 Gbps Myri-10GB offers performance and cost advantages over 10 GB over fiber Ethernet. It uses the same physical layers, and is highly interoperable with 10 GB over fiber Ethernet.
Chapter 10. Network subsystem
289
10.5.2 InfiniBand InfiniBand is a new architecture designed to provide an industry standard fabric for creating clusters that address HPC requirements, such as those found in scientific, technical, and financial applications. The InfiniBand high-bandwidth fabric permits high-speed interconnection between cluster servers. The combination of InfiniBand switches and InfiniBand host channel adapters (HCAs) means that a suitably configured cluster can deliver performance comparable to a single high-end server. In addition, InfiniBand can also be implemented within a single server by providing a high-speed interconnect between components. Uses of InfiniBand include the following: Application clustering, connecting servers together each running separate application instances but needing to share data quickly and efficiently Interprocess communications, enabling multiple servers to work together on a single application. SAN solutions, where the fabric topology of InfiniBand enables communication between storage and server to be simplified. InfiniBand is a switch-based serial I/O interconnect architecture operating at a base speed of 2.5 Gbps or 10 Gbps in each direction (per port). InfiniBand is a low pin count serial architecture that connects devices spanning distances up to 17 meters over ordinary twisted pair copper wires. Over fiber optic cable, it can span distances of several kilometers or more. Furthermore, InfiniBand provides both Quality of Service (QoS) and Reliability, Availability, and Serviceability (RAS) features.
290
Tuning IBM System x Servers for Performance
The InfiniBand System Fabric is shown in Figure 10-32.
Processor Node CPU
Processor Node
CPU
Mem
CPU
HCA
HCA
Switch
Switch
Switch
Switch
CPU
HCA
Mem
Switch
InfiniBand Fabric Storage Subsystem
I/O Chassis TCA Switch Controller TCA
TCA
TCA
I/O Module
I/O Module
I/O Module
Figure 10-32 InfiniBand System Fabric
The architecture defines a layered hardware protocol as well as a software layer to manage initialization and the communication between the devices. Each link can support multiple transport services for reliability and multiple prioritized virtual communication channels. To manage the communication within a subnet, the architecture defines a communication management scheme that is responsible for configuring and maintaining each of the InfiniBand elements. Management schemes are defined for error reporting, link failover, chassis management and other services, to ensure a solid connection fabric.
Chapter 10. Network subsystem
291
The InfiniBand feature set includes:
Layered protocol Packet-based communication Support for Quality of Service (QoS) Three link speeds – 1X - 2.5 Gbps, 4-wire – 4X - 10 Gbps, 16-wire – 12X - 30 Gbps, 48-wire Circuit board-direct, copper and Fiber Cable interconnects Subnet management protocol Remote DMA (RDMA) support Multicast and unicast support Reliable transport methods: message queuing Communication flow control: link level and end-to-end
InfiniBand not only delivers bandwidth, but also delivers the data directly to memory through RDMA transfers (see 10.3.6, “RDMA overview” on page 280). InfiniBand implements a reliable in-order transport connection in hardware and thus data is delivered extremely efficiently, with low latencies and without host CPU assistance. For more detail of InfiniBand technology, see IBM Eserver BladeCenter and Topspin InfiniBand Switch Technology, REDP-3949, which is available at: http://www.redbooks.ibm.com/abstracts/redp3949.html
292
Tuning IBM System x Servers for Performance
Part 3
Part
3
Operating systems The operating system is an integral part of the server. All applications and data have to pass through the operating system from the server hardware to the applications and back again. Just like the hardware and applications, you need to tune the operating system for performance. This part includes the following chapters: Chapter 11, “Microsoft Windows Server” on page 295 Chapter 12, “Linux” on page 371 Chapter 13, “VMware ESX Server” on page 425
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
293
294
Tuning IBM System x Servers for Performance
11
Chapter 11.
Microsoft Windows Server Windows Server 20031 is Microsoft’s mainstream server operating system and has been now for almost four years. Over previous versions of the Windows server operating system, including Windows 2000 Server and Windows NT Server, Windows Server 2003 offers considerable improvements in stability, performance, security, scalability and manageability. Since the last iteration of this chapter, Microsoft have announced three important enhancements to the core Windows 2003 server operating system. These are: Windows Server 2003, Service Pack 1 for 32-bit (x86) Editions Windows Server 2003, x64 (64-bit) Editions Windows Server 2003, Release 2 (R2) for 32-bit (x86) & 64-bit (x64) Editions This chapter builds upon the performance tuning techniques detailed in its previous release to also emphasize the performance benefits that can be realized from these important product releases.
1
Product screen captures and content reprinted with permission from Microsoft Corporation.
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
295
11.1 Introduction Windows Server 2003 is designed to be a largely “self-tuning” operating system. A standard “vanilla” installation of the operating system will yield sound performance results in most circumstances. In some instances however, specific settings and parameters can be tuned to optimize server performance even further. This chapter describes in detail many tuning techniques, any of which can become another weapon in your arsenal of methods to extract the best performance possible from your Windows server. Tip: As with all configuration changes, you need to implement the following suggestions one at a time to see what performance improvements are offered. If system performance decreases after making a change, then you need to reverse the change. Many of the changes listed in this chapter might only offer marginal performance improvements in and of themselves. The real benefits however of server tuning are realized when multiple tuning improvements are made and combined with one another. For a given server function, not all tuning techniques listed in this chapter will be appropriate. The challenge for the server engineer or architect is in determining which of these techniques, when combined, will yield the biggest performance enhancements. Many factors will play into this, including the server function, the underlying hardware and how this has been configured, and the network and storage infrastructure that the server is connected to. It is also well worth noting that some of the performance tuning techniques outlined in this chapter might no longer be relevant in the x64 (64-bit) versions of Windows Server 2003. Several of these techniques described throughout are used to tweak and work around the limitations of the x86 (32-bit) architecture. As a result, in the x64 versions, they are no longer relevant. Where known, this has been outlined.
The Windows Server 2003 family - 32-bit (x86) Editions Windows Server 2003 comes in four different 32-bit (x86) versions. These are:
Windows Server 2003, Web Edition Windows Server 2003, Standard Edition Windows Server 2003, Enterprise Edition Windows Server 2003, Datacenter Edition
Each of these has support for different hardware configurations and features and largely determines how scalable the server is. Table 11-1 on page 297 compares the capabilities of the various versions available in the 32-bit (x86) versions of Windows Server 2003.
296
Tuning IBM System x Servers for Performance
Note that the maximum amount of memory and the number of CPUs supported in the Enterprise and Datacenter editions of the 32-bit editions of Windows Server 2003 has increased with the release of Service Pack 1 (SP1) and Release 2 (R2). Table 11-1 Windows Server 2003 Family - 32-bit (x86) Editions
Requirement
Web Edition
Standard Edition
Enterprise Edition
Datacenter Edition
Maximum supported RAM
2 GB
4 GB
32/64* GB
64/128* GB
Number of supported processors
1 to 2
1 to 4
1 to 8
8 to 32/64** 8-way capable***
Server clustering
No
No
Up to 8 node
Up to 8 node
Support for /3GB switch
No
No
Yes
Yes
Support for /PAE switch
No
No
Yes
Yes
* Maximum physical memory (RAM) supported has increased from 32 GB to 64 GB for Enterprise Edition with R2 and from 64 GB to 128 GB for Datacenter Edition with R2. ** Maximum CPU support has increased from 32 to 64 CPUs for Datacenter Edition with R2 *** Windows Server 2003 Datacenter Edition requires a server that is eight-way capable but only requires a minimum of four-processors in the actual system.
The Windows Server 2003 family - 64-bit (x64) Editions Microsoft have not released the Web Edition of Windows Server 2003 in the 64-bit (x64) family of server operating systems. The editions that are available are:
Windows Server 2003, Standard x64 Edition Windows Server 2003, Enterprise x64 Edition Windows Server 2003, Enterprise Itanium Edition Windows Server 2003, Datacenter x64 Edition Windows Server 2003, Datacenter Itanium Edition
As it is a considerably later release, much of the code-base for the 64-bit (x64) editions of Windows Server 2003 are based on the same code the makes up the Service Pack 1 editions of Windows Server 2003. As a result, Service Pack 1 is not an option for the 64-bit (x64) editions. Release 2 (R2) is an optional extra for the 64-bit (x64) editions of Windows Server 2003, though it is expected that most
Chapter 11. Microsoft Windows Server
297
customers would install R2 by default to access the many extra features available within this latest product offering. Due to the fundamental architectural differences of 64-bit computing, vastly higher memory thresholds are available in the 64-bit (x64) editions of Windows Server 2003, as evidenced in Table 11-2. Table 11-2 Windows Server 2003 Family - 64-bit (x64) Editions
Requirement
Standard x64 Edition
Enterprise x64 Edition
Datacenter x64 Edition
Enterprise Itanium Edition
Datacenter Itanium Edition
Maximum supported RAM
32 GB*
1 TB
1 TB
1 TB
1 TB
Number of supported processors
1 to 4
1 to 8
8 to 64
1 to 8
8 to 64 8-way capable*
Server clustering
No
Up to 8 node
Up to 8 node
Up to 8 node
Up to 8 node
* Windows Server 2003 Datacenter Edition requires a server that is eight-way capable but only requires a minimum of four-processors in the actual system
A more thorough comparison of all feature differences between the various versions of the Windows Server 2003 operating system for both 32-bit and 64-bit editions can be found at: http://www.microsoft.com/windowsserver2003/evaluation/features/comparef eatures.mspx
11.2 Windows Server 2003, 64-bit (x64) Editions After years of working around the limitations of the 32-bit processor architecture, in April, 2005, Microsoft released the much awaited 64-bit editions of Windows Server 2003. While the Itanium version has been available for some time, it is the release of the editions of Windows Server 2003 to support the x64 processors that will see 64-bit computing finally transition into the Windows server mainstream. In a relatively short-period of time, it is expected that the 64-bit editions of Windows Server 2003 will displace the now seemingly archaic 32-bit cousin. This is largely assisted by the high-level of compatibility that Microsoft have built into the 64-bit (x64) operating system, offering true backward compatibility for 32-bit applications with little to no degradation in performance.
298
Tuning IBM System x Servers for Performance
11.2.1 32-bit limitations The amount of virtual memory that can be addressed by the 32-bit versions of Windows Server 2003 is 4 GB, through a virtual address space. On a standard implementation, this 4 GB is divided into 2 GB for kernel mode processes and 2 GB for application (user) mode processes. In Windows Server 2003, 32-bit editions, it is possible to increase the amount of memory available from 2 GB to 3 GB for 32-bit applications that have been designed to use more than 2 GB, through the use of the /3GB and /PAE switches, as explained in 11.13, “The /3GB BOOT.INI parameter (32-bit x86)” on page 334 and 11.14, “Using PAE and AWE to access memory above 4 GB (32-bit x86)” on page 335. This increase of available user memory from 2 GB to 3 GB presents problems, however: It imposes limitations on the amount of memory available to kernel mode processes to 1 GB It does not work around the architectural limit of the total 4 GB virtual address space. It increases the difficulty of developing applications as they need to make use of the Addressable Windows Extensions (AWE) application programming interface (API) to take advantage of Physical Address Extensions (PAE). It has not removed the physical memory constraint of 64 GB. With the upgrade to Windows Server 2003 64-bit (x64) editions, these limitations no longer exist and there are opportunities for significant improvements in server performance.
11.2.2 64-bit Benefits The single biggest performance increase of the 64-bit architecture is the amount of memory that can now be addressed. With Windows Server 2003 x64 Editions, the addressable virtual memory space increases from the 32-bit limit of just 4 GB to 16 TB. Entire databases, data sets, directory stores, indexes, Web caches and applications can now be loaded completely into memory, delivering often staggering processing performance improvements and vastly increased scalability. It is worth noting that the current Windows Server 2003 x64 editions actually only use 40 bits for addressing memory, offering an address space of 2^40, or 16 TB. 16 Exabytes is that actual theoretical maximum of a full 64-bit address space.
Chapter 11. Microsoft Windows Server
299
This virtual address space is divided evenly between user mode and kernel mode, as with 32-bit Windows. This provides native 64-bit applications with 8 TB of virtual address space. Even 32-bit applications that have been written to take advantage of memory greater than the 2 GB limitation of the 32-bit architecture can benefit from this immediately in that they can now address 4 GB of virtual address space as this space no longer needs to be shared with kernel mode processes. Table 11-3 highlights some of the key difference between 32-bit and 64-bit memory limits. Each of the notable improvements in these memory limits for the 64-bit (x64) platform offers real scalability and performance enhancements. Table 11-3 Memory limitations of 32-bit (x86) and 64-bit (x64) Windows Server 2003
Memory Limit
32-bit (x86)
64-bit (x64)
Total virtual address space
4 GB
16 TB
Virtual address space per 32-bit process
2 GB
4 GB
Virtual address space per 64-bit process
Not applicable
8 TB
Paged Pool
491 MB
128 GB
Non-paged Pool
256 MB
128 GB
System Page Table Entry (PTE)
660 MB to 990 MB
128 GB
The Windows on Windows 64 emulator (WOW64) allows 32-bit applications to run on Windows Server 2003 x64 Editions exactly as they might run on a 32-bit edition. This has been written so optimally however than any overhead imposed is in emulation activities is very marginal and in many cases imperceptible. Even with the emulation between 32-bit and 64-bit, in several cases 32-bit applications will run faster on Windows Server 64-bit (x64) Editions due to other improvements in the operating system, offering another notable benefit to this new operating system version. One of the other most notable advantage of the 64-bit (x64) editions is the greatly increased amount of physical RAM than can be supported, offering huge scalability benefits. With the Standard Edition of Windows Server 2003 x64, the maximum supported memory is 32 GB. With the Enterprise and Datacenter Editions however, this blows out to a considerably larger 1 TB of RAM. When compared to the previous memory maximums of 4 GB, 64 GB and 128 GB of the Standard, Enterprise and Datacenter editions of Windows Server 2003 R2, the increase in supportable memory, and hence performance, is significant.
300
Tuning IBM System x Servers for Performance
Overall performance improvements in disk and network I/O throughput and efficiency should be evident in the 64-bit (x64) editions of Windows Server 2003. Greater physical memory support and addressable memory space means that caches can be considerably larger, improving I/O performance. An increased number of larger (wider) processor registers will also deliver notable performance gains to applications as data does not need to be written out to memory as frequently and function calls can process more arguments. Windows Server 2003 64-bit (x64) Editions also delivers improved reliability over previous versions. Based on the exactly the same source code as Windows Server 2003 Service Pack 1 32-bit editions (x86), the newer edition will offer the same reliability that this platform has offered to date. In addition, Windows Server 2003 64-bit (x64) editions include Patch-Guard, a technology which protects poorly-written code third-party code from patching the Windows kernel which in turn could destabilize or crash a server. Security improvements are also available with the 64-bit (x64) edition of Windows Server 2003. Building on the benefits of Data Execution Prevention (DEP) released first in Windows Server 2003, Service Pack 1 (32-bit x86), the 64-bit (x64) editions all include this feature as a standard. DEP prevents Windows against buffer overflows, effectively stopping malicious code from being able to execute from memory locations it should not. All these features of Windows Server 2003 64-bit (x64) editions serve to make this new version of the operating system the most high-performing, scalable, stable and secure version released to date.
11.2.3 The transition to 64-bit computing With the release of Intel 64 Technology (previously known as EMT64T) and AMD AMD64, server hardware vendors have made the transition from 32-bit (x86) to 64-bit (x64) processing a very straightforward process. These processors support both 32-bit and 64-bit operating systems and applications, making the migration path an easy one. With the 64-bit (x64) versions of Windows Server 2003 able to run 32-bit applications directly, often at a much higher level of performance, the move to the optimal native 64-bit computing should present few hurdles.
Chapter 11. Microsoft Windows Server
301
11.2.4 Acknowledgements Much of this material for this section on Windows Server 2003 64-bit (x64) editions has been collated from two key articles available from the Microsoft Web site. More detail and case studies of the benefits of Windows Server 2003 64-bit (x64) computing can be found by referring to these two papers: Benefits of Microsoft Windows x64 Editions http://www.microsoft.com/windowsserver2003/techinfo/overview/x64bene fits.mspx Windows Server 2003 x64 Editions Deployment Scenarios http://www.microsoft.com/windowsserver2003/64bit/x64/deploy.mspx
11.3 Windows Server 2003, Release 2 (R2) Windows Server 2003, R2 is an update release for the Windows Server 2003 operating system. This release brings an impressive array of additional features to the native operating system. This release is different from a Service Pack in that it brings new features and functionality to the operating system while as Service Pack is a rollup of fixes, updates and patches at a given point in time. That said, the installation of R2 is dependent on Windows Server 2003, Service Pack 1 already being installed. R2 offers enhancements to Windows Server 2003 in the following main areas:
Simplified branch office server management Improved identity and access management Reduced storage management costs Rich Web platform Cost effective server virtualization Seamless UNIX / Windows Interoperability
For more detail on the features delivered by R2, visit the following links: http://www.microsoft.com/windowsserver2003/R2/whatsnewinr2.mspx http://download.microsoft.com/download/7/f/3/7f396370-86ba-4cb5-b19e-e7 e518cf53ba/WS03R2RevGuide.doc The components of R2 that offer notable performance benefits are those included to improve branch office server manageability, such as the following: Robust file replication The replication engine for the Distributed File System (DFS™) has been completely rewritten in Windows Server 2003 R2. DFS is multimaster file
302
Tuning IBM System x Servers for Performance
replication service, significantly more scalable and efficient in synchronizing file servers than File Replication Services (FRS), its predecessor. DFS schedules and throttles replication processes, supports multiple replication topologies, and utilizes Remote Differential Compression (RDC) to increase WAN efficiency. If WAN connections fail, data can be stored and forwarded when WAN connections become available. Through the efficiency gains of these new features in R2 DFS, the performance of core user-facing processes improves. Advanced compression technologies Remote Differential Compression (RDC) is a WAN-friendly compression technology that replicates only the changes needed to ensure global file consistency.Any WAN performance improvements often serve to improve the user experience.
11.4 Processor scheduling Windows uses preemptive multitasking to prioritize process threads that the CPU has to attend to. Preemptive multitasking is a methodology whereby the execution of a process is halted and another is started, at the discretion of the operating system. This prevents a single thread from monopolizing the CPU. Switching the CPU from executing one process to the next is known as context-switching. The Windows operating system includes a setting that determines how long individual threads are allowed to run on the CPU before a context-switch occurs and the next thread is serviced. This amount of time is referred to as a quantum. This setting lets you choose how processor quanta are shared between foreground and background processes. Typically for a server, it is not desirable to allow the foreground program to have more CPU time allocated to it than background processes. That is, all applications and their processes running on the server should be given equal contention for the CPU. We recommend selecting Background services so that all programs receive equal amounts of processor time. To change this: 1. 2. 3. 4.
Open the System Control Panel. Select the Advanced tab. Within the Performance frame, click Settings. Select the Advanced tab. The window shown in Figure 11-1 opens.
Chapter 11. Microsoft Windows Server
303
This setting is the preferred setting for most servers.
Figure 11-1 Configuring processor scheduling
Modifying the value using the control panel applet as described above modifies the following registry value to affect the duration of each quanta: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\PriorityControl\ Win32PrioritySeparation The Win32PrioritySeparation Registry values in Windows Server 2003 are: 0x00000026 (38) for best performance of Programs 0x00000018 (24) for best performance of Background services. These values remain the same between the 32-bit (x86) and 64-bit (x64) editions of the Windows Server 2003 operating system. We strongly recommend you use only the control panel applet shown in Figure 11-1 for these settings in order to always get valid, appropriate, operating system revision-specific, and optimal values in the registry.
304
Tuning IBM System x Servers for Performance
11.5 Virtual memory Memory paging occurs when memory resources required by the processes running on the server exceed the physical amount of memory installed. Windows, like most other operating systems, employs virtual memory techniques that allow applications to address greater amounts of memory than what is physically available. This is achieved by setting aside a portion of disk for paging. This area, known as the paging file, is used by the operating system to page portions of physical memory contents to disk, freeing up physical memory to be used by applications that require it at a given time. The combination of the paging file and the physical memory installed in the server is known as virtual memory. Virtual memory is managed in Windows by the Virtual Memory Manager (VMM). Physical memory can be accessed at exponentially faster rates than disk. Every time a server has to move data between physical memory and disk will introduce a significant system delay. While some degree of paging is normal on servers, excessive, consistent memory paging activity is referred to as thrashing and can have a very debilitating effect on overall system performance. Thus, it is always desirable to minimize paging activity. Ideally servers should be designed with sufficient physical memory to keep paging to an absolute minimum. The paging file, or pagefile, in Windows, is named PAGEFILE.SYS. Virtual memory settings are configured through the System control panel. To configure the page file size: 1. 2. 3. 4. 5.
Open the System Control Panel. Select the Advanced tab. Within the Performance frame, click Settings. Select the Advanced tab. Click Change. The window shown in Figure 11-2 opens.
Windows Server 2003 has several options for configuring the page file that previous versions of Windows did not. Windows Server 2003 has introduced new settings for virtual memory configuration, including letting the system manage the size of the page file, or to have no page file at all. If you let Windows manage the size, it will create a pagefile of a size equal to physical memory + 1 MB. This is the minimum amount of space required to create a memory dump in the event the server encounters a STOP event (blue screen).
Chapter 11. Microsoft Windows Server
305
Create separate pagefiles on multiple physical drives to improve system performance.
Figure 11-2 Virtual memory settings
A pagefile can be created for each individual volume on a server, up to a maximum of sixteen page files and a maximum 4 GB limit per pagefile. This allows for a maximum total pagefile size of 64 GB. The total of all pagefiles on all volumes is managed and used by the operating system as one large pagefile. When a pagefile is split between smaller pagefiles on separate volumes as described above, when it needs to write to the pagefile, the virtual memory manager optimizes the workload by selecting the least busy disk based on internal algorithms. This ensures best possible performance for a multiple-volume pagefile. While not best practice, it is possible to create multiple page files on the same operating system volume. This is achieved by placing the page files in different folders on the same volume. This change is carried out through editing the system registry rather than through the standard GUI interface. The process to achieve this is outlined in Microsoft KB article 237740: http://support.microsoft.com/?kbid=237740 We do not recommend this approach as no performance gain will be achieved by splitting the page file into segments on the same volume regardless of the underlying physical disk or array configuration.
306
Tuning IBM System x Servers for Performance
11.5.1 Configuring the pagefile for maximum performance gain Optimal pagefile performance will be achieved by isolating pagefiles to dedicated physical drives running on RAID-0 (striping) or RAID-1 (mirroring) arrays, or on single disks without RAID at all. Redundancy is not normally required for pagefiles, though performance might be improved through the use of some RAID configurations. By using a dedicated disk or drive array, this means PAGEFILE.SYS is the only file on the entire volume and risks no fragmentation caused by other files or directories residing on the same volume. As with most disk-arrays, the more physical disks in the array, the better the performance. When distributed between multiple volumes on multiple physical drives, the pagefile size should be kept uniform between drives and ideally on drives of the same capacity and speed. We strongly recommend against the use of RAID-5 arrays to host pagefiles as pagefile activity is write intensive and thus not suited to the characteristics of RAID-5. Where pagefile optimization is critical, do not place the pagefile on the same physical drive as the operating system, which happens to be the system default. If this must occur, ensure that the pagefile exists on the same volume (typically C:) as the operating system. Putting it on another volume on the same physical drive will only increase disk seek time and reduce system performance as the disk heads will be continually moving between the volumes, alternately accessing the page file, operating system files and other applications and data. Remember too that the first partition on a physical disk is closest to the outside edge of the physical disk, the one typically hosting the operating system, where disk speed is highest and performance is best. Note if you do remove the paging file from the boot partition, Windows cannot create a crash dump file (MEMORY.DMP) in which to write debugging information in the event that a kernel mode STOP error message (“blue screen of death”) occurs. If you do require a crash dump file, then you will have no option but to leave a page file of at least the size of physical memory + 1 MB on the boot partition. We recommend setting the size of the pagefile manually. This normally produces better results than allowing the server to size it automatically or having no page file at all. Best-practice tuning is to set the initial (minimum) and maximum size settings for the pagefile to the same value. This ensures that no processing resources are lost to the dynamic resizing of the pagefile, which can be intensive. This is especially given this resizing activity is typically occurring when the memory resources on the system are already becoming constrained. Setting the same minimum and maximum page file size values also ensures that the paging area on a disk is one single, contiguous area, improving disk seek time.
Chapter 11. Microsoft Windows Server
307
Windows Server 2003 automatically recommends a total paging file size equal to 1.5 times the amount of installed RAM. On servers with adequate disk space, the pagefile on all disks combined should be configured up to twice (that is, two times) the physical memory for optimal performance. The only drawback of having such a large pagefile is the amount of disk space consumed on the volumes used to accommodate the page file(s). Servers of lesser workloads or those tight on disk space should still try to use a pagefile total of at least equal to the amount of physical memory.
11.5.2 Creating the pagefile to optimize performance Creating the whole pagefile in one step reduces the possibility of having a partitioned pagefile and therefore improves system performance. The best way to create a contiguous static pagefile in one step is to follow this procedure for each pagefile configured: 1. Remove the current page files from your server by clearing the Initial and Maximum size values in the Virtual Memory settings window or by clicking No Paging File, then clicking Set (Figure 11-2 on page 306). 2. Reboot the machine and click OK. Ignore the warning message about the page file. 3. Defragment the disk you want to create the page file on. This step should give you enough continuous space to avoid partitioning of your new page file. 4. Create a new page file with the desired values as described is 11.5.2, “Creating the pagefile to optimize performance” on page 308. 5. Reboot the server. An ever better approach is to re-format the volume entirely and create the pagefile immediately before placing any data on the disk. This ensures the file is created as one large contiguous file as close to the very outside edge of the disk as possible, ensuring no fragmentation and best disk access performance. The work and time involved in moving data to another volume temporarily to achieve this outcome often means, however, that this procedure is not always achievable on a production server.
11.5.3 Measuring pagefile usage A good metric for measuring pagefile usage is Paging file: %Usage Max in the Windows System Monitor. If this reveals consistent use of the page file, then consider increasing the amount of physical memory in the server by this amount. For example, if a pagefile is 2048 MB (2 GB) and your server is consistently showing 10% usage, it would be prudent to add an additional say 256 MB RAM.
308
Tuning IBM System x Servers for Performance
While today it is often considered an easy and relatively inexpensive way to upgrade a server and improve performance, adding additional memory should only be done after investigating all other tuning options and it has been determined that memory is indeed the system bottleneck. There is simply no benefit to be gained by having more physical memory in a server than it can put to use. Additional, unused memory might be better put to use in another server.
11.6 File system cache The file system cache is an area of physical memory set aside to dynamically store recently accessed data that has been read or written to the I/O subsystem, including data transfers between hard drives, networks interfaces, and networks. The Windows Virtual Memory Manager (VMM) copies data to and from the system cache as though it were an array in memory. The file system cache improves performance by reducing the number of accesses to physical devices attached to the I/O subsystem of the server. By moving commonly used files into system cache, disk and network read and write operations are reduced and system performance is increased. You can optimize Windows server performance by tuning the file system cache. Performance of the file system cache is greatly improved in the 64-bit (x64) editions of Windows Server 2003. The default 2 GB kernel maximum virtual memory address space in the 32-bit (x86) editions of Windows is a major bottleneck as this same space is shared by the system page table entries (PTE), page pool memory, non-paged pool memory and file system cache. Using the /3GB switch on 32-bit (x86) systems can improve application performance (as described in 11.9, “Optimizing the protocol binding and provider order” on page 320 and 11.10, “Optimizing network card settings” on page 322) but forces the Windows kernel to operate in only 1 GB of virtual address space, potentially making the situation worse. These same constraints no longer apply in the 64-bit (x64) editions of Windows, greatly enhancing system performance. Tip: The method for managing the file system cache has changed in Windows Server 2003. There are now two applets, not one as in previous versions of Windows. In previous versions of the Windows server operating system, one Control Panel applet was used for managing the file system cache. Now, in Windows Server 2003, two configuration options exist that determine how much system memory is available to be allocated to the working set of the file system cache versus how much memory is able to be allocated to the working set of applications, and the priority with which they are managed against one another.
Chapter 11. Microsoft Windows Server
309
The selection made in these dialogs will depend on the intended server function. The configuration options are: File and Printer Sharing for Microsoft Networks (Network Control Panel applet) System (System Control Panel applet), also referred to in Microsoft documentation as “Performance Options” as it has consolidated several performance applets into one location. The File and Printer Sharing dialog can be accessed as follows: 1. Click Start → Control Panel → Network Connections. 2. While still in the Start menu context, right-click Network Connections and choose Open. 3. Select any of Local Area Connections including any teamed network interface. This setting affects all LAN interfaces, so which LAN connection you choose in the above steps is not important. 4. Right-click the selected connection object and choose Properties. 5. Select File and Printer Sharing for Microsoft Networks. 6. Click Properties. The window that is shown in Figure 11-3 opens.
310
Tuning IBM System x Servers for Performance
Better for file servers and servers with amounts of physical RAM typically exceeding 2 GB
Better for application servers and those with internal memory management features
Figure 11-3 File and Print Sharing for Microsoft Networks applet: server optimization
The four options here modify two registry entries: HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\Size HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\LargeSystemCache The value of the registry entries will be set depending on the option selected in the control panel as shown in Table 11-4. Table 11-4 Registry effect of Server Optimization option selected
Server optimization option selected
LanmanServer Size
LargeSystemCache
Minimize memory used
1
0
Balance
2
0
Maximize data throughput for file sharing
3
1
Maximize data throughput for network applications
3
0
Chapter 11. Microsoft Windows Server
311
These values are the same for both the 32-bit (x86) and 64-bit (x64) editions of Windows Server 2003. The file system cache has a working set of memory like any other process. The option chosen in this dialog effectively determines how large the working set is allowed to grow to and with what priority the file system cache is treated by the operating system relative to other applications and processes running on the server. Typically only one of the bottom two options in the control panel is employed for an enterprise server implementation and are thus the only two detailed here: Maximize throughput for file sharing This option is the default setting. It instructs the operating system to give the working set of the file system cache a higher priority for memory allocation than the working sets of applications. It will yield the best performance in a file server environment that is not running other applications. if other applications are running on the server, it will require sufficient physical memory to obtain the maximum performance gain as more memory is set aside for the file system cache than for applications. As a result the “maximize throughput for network applications” is typically used. This “file sharing” option might, however, be the best option to use on servers with large quantities of physical memory as described below. Maximize throughput for network applications This choice is the recommended setting for machines running applications that are memory-intensive. With this option chosen, the working set of applications will have a priority over the working set of the file system cache. This setting is normally the best setting to use for all servers except those with (a) dedicated file servers or with applications exhibiting file server-like characteristics or (b) those with significant amounts of memory (see below). The second control panel used for managing the file system cache in Windows Server 2003 is within the System applet: 1. Click Start → Control Panel → System. 2. Select the Advanced tab. 3. Within the Performance frame, click Settings. 4. Select the Advanced tab. The window shown in Figure 11-4 opens.
312
Tuning IBM System x Servers for Performance
Memory optimization settings for the file system cache are controlled here
Figure 11-4 System applet: memory usage
The System applet reflects and modifies the value included within the same LanmanServer “LargeSystemCache” registry key as described above for File and Printer Sharing in the Network applet in Figure 11-3 on page 311. Making a change to the LargeSystemCache through this applet however does so without impacting the MemoryManagement “Size” value that File and Printer Sharing does. Given that most users only use the Maximize throughput for network applications or the Maximize throughput for file sharing options for enterprise servers, the Size value remains same, a value of 3. This setting means that using the System applet to adjust the LargeSystemCache value is redundant as it is just as easily set using File and Print Sharing. As a result we recommend using the first control panel as described above and leave this second control panel untouched. It would seem that the only advantage to using both Control Panel applets in conjunction would allow you to have the applets actually indicate Maximize throughput for network applications and simultaneously indicate memory usage favors System cache. This same effect to the registry is achieved by selecting Maximize throughput for file-sharing (as per Table 11-4 on
Chapter 11. Microsoft Windows Server
313
page 311)—visually, it simply does not say “Maximize throughput for network applications.” If you do desire this change purely for aesthetic reasons then make sure you set the first Network applet before the second System applet as the first overrides the second selections, but the reverse does not occur.
11.6.1 Servers with large amounts of free physical memory How much memory is able to be allocated to the file system cache depends on how much physical memory exists in the server and the file system cache option selected in the dialogs above. With Windows Server 2003, when Maximize data throughput for file sharing is selected (LargeSystemCache set to “1”), the maximum the file system cache can grow to is 960 MB. When Maximize throughput for network applications is selected (LargeSystemCache set to “0”) the maximum the file system cache can grow to is 512 MB. (See Microsoft KB 837331, URL below) Depending on the selection made here it is possible that adding more physical memory up to point will allow the file system cache to grow even larger, up to these stated maximums. On a server with physical memory from say 2 GB and upwards, it might be preferable to leave Maximize data throughput for file sharing option selected. That is, providing the total amount of memory used by the operating system and server applications does not exceed the amount of physical RAM minus 960 MB. In fact any application server that can be determined to have 960 MB or more of RAM unused will likely be given a performance boost by enabling the large system cache. By enabling this, all the disk and network I/O performance benefits of using a large file system cache are realized and the applications running on the server continue to run without being memory constrained. Some applications have their own memory management optimizers built into them, including Microsoft SQL Server and Microsoft Exchange. In such instances, the setting above is best set to Maximize throughput for network applications and let the application manage memory and their own internal system cache as it sees appropriate. See Microsoft Knowledge Base article 837331 for more information: http://support.microsoft.com/?kbid=837331 Note well that the maximum size of the file system cache increases from 960 MB in the 32-bit (x86) edition of Windows Server 2003 to 1 TB in the 64-bit (x64) editions. This has potential to yield enormous performance improvements on systems where the file system cache is actively used.
314
Tuning IBM System x Servers for Performance
11.7 Disabling or removing unnecessary services When Windows is first installed, many services are enabled that might not be necessary for a particular server. While in Windows Server 2003 many more services are disabled by default than in previous editions of the server operating system, there still remains on many systems an opportunity for improving performance further by examining running services. Inexperienced users might also inadvertently add additional services when installing or updating the operating system that are not actually required for a given system. Each service requires system resources and as a result is best to disable unnecessary services to improve performance. Care does need to be taken when disabling services. Unless you are completely certain of the purpose of a given service it is recommended to research it further before choosing to disable it. Disabling some services that the operating system requires to be running can render a system inoperable and possibly unable to boot. To view the services running in Windows, complete the following steps: 1. Right-click My Computer and select Manage. 2. Expand the Services and Applications icon. 3. Select the Services icon. 4. Click the Standard tab at the bottom of the right-pane. A window similar to Figure 11-5 on page 316 opens. All the services installed on the system are displayed. The status, running or not, is shown in the third column. 5. Click twice on Status at the top of the third column shown. This sorts together all running (Started) services from those that are not running.
Chapter 11. Microsoft Windows Server
315
Figure 11-5 Windows Services
From this dialog, all services that are not required to be running on the server should be stopped and disabled. This will prevent the service from automatically starting at system boot time. To stop and disable a service, do the following: 1. Right-click the service and click Properties. 2. Click Stop and set the Startup type to Disabled. 3. Click OK to return to the Services window. If a particular service has been installed as part an application or Windows component and is not actually required on a given server, a better approach is to remove or uninstall this application or component altogether. This is typically performed through the Add or Remove Programs applet in Control Panel. Some services might not be required at system boot time but might be required to start by other applications or services at a later time. Such services should be set to have a startup type of Manual. Unless a service is explicitly set to have a
316
Tuning IBM System x Servers for Performance
startup type of Disabled, it can start at any time and perhaps unnecessarily use system resources. Windows Server 2003 comes installed with many services that Windows 2000 Server and Windows NT Server did not. Designed as a significantly more secure operating system than its predecessors, many of the services have their startup type set to Disabled or Manual by default. Nonetheless, there remains several services enabled on a standard installation that can likely be disabled on many servers. For example, the Print Spooler service is enabled by default but is not usually required if the server is not functioning as a print server or does not have local printing requirements. Table 11-5 lists services on a standard Windows Server 2003 installation that should be reviewed for their requirement on your systems. This is not a definitive list of all services, just those that should be considered for their applicability on an enterprise server. Note well that this list includes only those services that are not already disabled by default on Windows Server 2003 and might be candidates for disabling. These services might still be required for your environment, depending on the particular function of the server and the applications it is running. For example, the File Replication service (FRS) is normally required on an Active Directory domain controller, but its inclusion with other server types should be questioned. Each server is different and implementing the following recommendations should be tested before changing. Table 11-5 Windows service startup recommendations
Service
Default startup type
Recommended setting
Application Management
Manual
Disabled
Alerter
Automatic
Disabled
Clipbook
Disabled
Disabled
Computer Browser
Automatic
Disabled
Distributed file system
Automatic
Disabled
Distributed link tracking client
Automatic
Disabled
Distributed transaction coordinator
Automatic
Manual
Error Reporting Service
Automatic
Disabled
Fax Service
Manual
Disabled
File Replication
Manual
Disabled
Chapter 11. Microsoft Windows Server
317
Service
Default startup type
Recommended setting
Help and Support
Automatic
Disabled
HTTP SSL
Manual
Disabled
License Logging
Manual
Disabled
Logical Disk Manager
Automatic
Manual
Messenger
Automatic
Disabled
Portable Media Serial Number Service
Manual
Disabled
Shell Hardware Detection
Automatic
Disabled
Windows Audio
Automatic
Disabled
Wireless Configuration
Automatic
Disabled
11.8 Removing unnecessary protocols and services Windows servers often have more network services and protocols installed than are actually required for the purpose or application for which they have been implemented. Each additional network client, service or protocol places additional overhead on system resources. In addition, each protocol generates network traffic. By removing unnecessary network clients, services and protocols, system resources are made available for other processes, excess network traffic is avoided and the number of network bindings that must be negotiated is reduced to a minimum. TCP/IP is largely viewed as the de facto enterprise network protocol in modern networks. Unless integration with other systems is required it is likely sufficient now to just have TCP/IP loaded as the only network protocol on your server. To view the currently installed network clients, protocols and services: 1. Click Start → Control Panel → Network Connections. 2. While still in the Start menu context, right-click Network Connections and choose Open. 3. Click Properties. 4. Right-click Local Area Connection (or the entry for your network connection). 5. Click Properties. The window shown in Figure 11-6 opens.
318
Tuning IBM System x Servers for Performance
Temporarily disable a component by removing the tick from the adjacent check box.
Remove an unnecessary component altogether by selecting the item and clicking Uninstall.
Figure 11-6 Network clients, services and protocols
To remove an unnecessary item, select it and click Uninstall. To disable the item temporarily without completely uninstalling it, simply remove the tick from the check box beside it. This latter approach (disable rather than uninstall) might be a more appropriate method in determining which services, protocols and clients are actually required on a system. When is has been determined that disabling an item has no adverse affect on the server, it can then be uninstalled. In many instances, the three components listed in Figure 11-6 are often sufficient for a file and print server on a standard TCP/IP based network. That is: Client for Microsoft Networks File and Printer Sharing for Microsoft Networks Internet Protocol (TCP/IP)
Chapter 11. Microsoft Windows Server
319
11.9 Optimizing the protocol binding and provider order Optimizing the protocol order and the provider order can also make a difference to performance.
Protocol binding order On a system supporting more than one network protocol, the order in which they are bound to the network clients and services running on the server is important. All network communications for a given service or client start with the protocol listed at the top of the binding list. If after a given period, no response is received, communications are routed to the next protocol in the list until all protocols are exhausted. As a result it is crucial to ensure the most frequently used protocol for a given client or service is moved to the top of the binding list to offer the best network I/O performance possible. To view the order of network bindings, do the following: 1. Click Start → Control Panel → Network Connections. 2. While still in the Start menu context, right-click Network Connections and choose Open. 3. Click Properties. 4. From the menu bar, click Advanced → Advanced Settings. The window shown in Figure 11-7 opens.
320
Tuning IBM System x Servers for Performance
Select a protocol and click either up or down to change the protocol binding. Disable the selected binding be removing the check box adjacent to the protocol.
Figure 11-7 Windows protocol binding order
By selecting a protocol and clicking the up and down buttons, you can change the binding priority of your protocols. If an installed protocol is not required by a particular service or client, it should be disabled. Do so by removing the tick in the check box beside the protocol in question. This will improve system performance and possibly improve security.
Network and print provider order Servers will often have multiple network and print providers installed. Similar to network bindings, the order in which these are configured will determine how quickly they respond to client requests for services running on the server. It will also affect how quickly the server itself connects to hosts when functioning as a client. The most commonly used network providers should be moved to the top of the list with the remaining ones ordered down in order of decreasing priority. To access the network provider order configuration: 1. Click Start → Control Panel → Network Connections. 2. While still in the Start menu context, right-click Network Connections and choose Open.
Chapter 11. Microsoft Windows Server
321
3. Click Properties. 4. From the menu bar, click Advanced → Advanced Settings. 5. Select the Network Provider tab. The window shown in Figure 11-8 opens.
Select a network or print provider and click either up or down to change the priority order,
Figure 11-8 Windows network provider order
By selecting a network or print provider and clicking the up and down buttons, you can change the order in which the computer responds to client requests.
11.10 Optimizing network card settings Many network interface cards in servers today have settings that can be configured through the Windows interface. Setting these optimally for your network environment and server configuration can significantly affect the performance of network throughput. Of all the performance tuning features outlined in this chapter, it is the ones in this section that have been noted to have the biggest improvement on system performance and throughput.
322
Tuning IBM System x Servers for Performance
To access this range of settings, following these steps: 1. Click Start → Settings → Network Connections. 2. Click Properties. 3. Right-click Local Area Connection (or the name of your network connection). 4. Click Properties. The window shown in Figure 11-9 opens.
Click Configure to access the configuration settings available for the network interface card.
Figure 11-9 Accessing the network interface card configuration
5. Click Configure.
Chapter 11. Microsoft Windows Server
323
6. Click the Advanced tab. A dialog box similar to that in Figure 11-10 opens, depending on the network adapter your system is using.
Figure 11-10 Network interface card advanced settings configuration
The exact configuration settings available differ from one network interface card to another. However, a handful of settings are common between most Intel-based cards in the IBM System x range of servers. Note: You apply these settings for each physical network interface, including the individual cards within a set teamed of interfaces that are configured for aggregation, load balancing, or fault tolerance. With some teaming software, you might need to apply these settings to the team also. Note also that some network interface cards are largely self-tuning and do not offer the option to configure parameters manually.
324
Tuning IBM System x Servers for Performance
The following settings are the ones that can have the most dramatic impact to performance: Link Speed and Duplex Experience suggests that the best practice for setting the speed and duplex values for each network interface in the server is to configure them in one of two ways: – Set to auto-negotiation if, and only if, the switch port is also set to auto negotiation also. The server and switch should then negotiate the fastest possible link speed and duplex settings. – Set to the same link speed and same duplex settings as those of the switch. These settings will, of course, normally yield the best performance if set to the highest settings that the switch will support. We do not recommend the use of auto-negotiation the server network interface combined with manually setting the parameter on the switch, or vice-versa. Using such a combination of settings at differing ends of the network connection to the server has often found to be the culprit of poor performance and instability in many production environments and should definitely be avoided. To repeat, use either auto-negotiation at both interfaces, or hard-code the settings at both interfaces, but not a mix of both of these. For more information, see the following Cisco Web site: http://www.cisco.com/warp/public/473/46.html#auto_neg_valid Receive Buffers This setting specifies the number of memory buffers used by the network interface driver when copying data to the protocol memory. It is normally set by default to a relatively low setting. We recommend setting this value as high as possible for maximum performance gains. On servers low on physical memory, this can have a negative impact as these buffers are taken from available physical memory on the system. On most modern systems however, the maximum setting can be implemented without any notable impact to memory resources. The amount of memory used by modifying this setting can easily be determined by watching the appropriate metrics in the Task Manager or System Monitor before and after making the changes. Monitor this impact before making the change permanent. Coalesce Buffers Map registers are system resources used in physical to virtual address conversion with bus mastering cards like the ones in some IBM System X servers. Coalesce buffers are those available to the network driver if the
Chapter 11. Microsoft Windows Server
325
driver runs out of map registers. We recommend setting this value as high as possible for maximum performance gains. Note the same impact to memory as with receive buffers is possible when increasing the number of coalesce buffers. Transmit Descriptors / Transmit Control Blocks This setting specifies how many transmit control buffers the driver allocates for use by the network interface. This directly reflects the number of outstanding packets the driver can have in its “send” queue. We recommend setting this value as high as possible for maximum performance gains. Note the same impact to memory as with receive buffers is possible when increasing the number of transmit descriptors / transmit control blocks. Offload features In almost all instances there will be benefit derived from enabling network interface offload features. In some instances the network interface might not be able to handle the offload capabilities at high throughput however as a general rule enabling offload his will benefit overall system performance. Some network interfaces have separate options or parameters to enable or disable offloading for send and receive traffic. Other advanced settings are often available with network interface cards than just those described here. The documentation for the network interface should be consulted to detail the meaning and impact of changing each setting. Where possible, use these settings to take network processing requirements away from the server itself and to the network interface. That is, “offload” the network requirements from the server CPU where possible and try to have the network interface do as much of the processing as it can. This will ensure optimal performance.
11.11 Process scheduling, priority levels, and affinity The scheduler is a component of the Windows operating system kernel. It is the scheduler that coordinates the servicing of the processes and their threads waiting and ready to use the system CPUs. The kernel schedules ready threads based upon their individual dynamic priority. The dynamic priority is a number between 0 and 31 that determines the importance of threads relative to one another. The higher the priority value, the higher the priority level. For example, a thread with a priority of 15 is serviced more quickly than a thread with a priority of 10. Even if it requires preempting a thread of lower priority, threads with the highest priority always run on the processor. This activity ensures Windows still pays
326
Tuning IBM System x Servers for Performance
attention to critical system threads required to keep the operating system running. A thread will run on the processor for either the duration of its CPU quantum (or time slice, described in 11.2, “Windows Server 2003, 64-bit (x64) Editions” on page 298) or until it is preempted by a thread of higher priority. Task Manager allows you to easily see the priority of all threads running on a system. To do so, open Task Manager, and click View → Select Columns, then select Base Priority, as shown in Figure 11-11.
Select Base Priority to ensure that you can see the priority of all running processes.
Figure 11-11 Selecting Base Priority in Task Manager
Chapter 11. Microsoft Windows Server
327
This displays a column in Task Manager as shown in Figure 11-12 that allows you to see the relative priority of processes running on the system.
The Base Priority column shows process priority values relative to each other.
Figure 11-12 Windows Task Manager displaying the Base Priority column
Most applications loaded by users run at a normal priority which has a base priority value of 8. Task Manager also allows the administrator the ability to change the priority of a process, either higher or lower. To do so, right-click the process in question, and click Set Priority from the drop-down menu as shown in Figure 11-13 on page 329. Then click the new priority that you want to assign to the process. Note: This procedure changes the priority of actual processes running on the system, but the change only lasts as long as the life of the selected process. If you want to launch a process with a non-normal priority, you can do so using the START command from a command-prompt. Type START /? for more information about how to do this.
328
Tuning IBM System x Servers for Performance
Task Manager allows you to change the priority of a given process from six options.
Figure 11-13 Changing the priority of a process using Task Manager
Threads, as a sub-component of processes, inherit the base priority of their parent process. The four priority classes are:
Idle Normal High Realtime
Each process’s priority class sets a range of priority values (between 1 and 31) and the threads of that process have a priority within that range. If the priority class is Realtime (priorities 16 to 31), the thread’s priority can never change while it is running. A single thread running at priority 31 will prevent all other threads from running. Conversely, threads running in all other priority classes are variable, meaning the thread’s priority can change while the thread is running. For threads in the Normal or High priority classes (priorities 1 through 15), the thread's priority can be raised or lowered by up to a value of 2 but cannot fall below its original, program-defined base priority.e priority
Chapter 11. Microsoft Windows Server
329
When should you modify the priority of a process? In most instances, you should do this as rarely as possible. Windows normally does a very good job of scheduling processor time to threads. Changing process priority is not an appropriate long-term solution to a bottleneck on a system. If you are suffering performance problems related to processes not receiving sufficient processing time, eventually additional or faster processors will be required to improve the situation. Normally the only conditions under which the priority of a process should be modified are when the system is CPU-bound. Processor utilization, queue length and context switching can all be measured using System Monitor to help identify processor bottlenecks. In a system with plenty of spare CPU capacity, testing has shown that changing the base priority of a process offers marginal, if any, performance improvements. This is because the processor is comfortable with the load it is under and able to schedule time appropriately to threads running on the system. Conversely, on a system suffering heavy CPU-load, CPU time being allocated to nominated processes will likely benefit from changing the base priority. On extremely busy systems, threads with the lowest priority will be serviced infrequently, if at all. Modifying process priorities can be an effective troubleshooting technique for solving short-term performance problems however it is rarely a long-term solution. Important: Changing priorities might destabilize the system. Increasing the priority of a process might prevent other processes, including system services, from running. In particular, be careful not to schedule many processes with the High priority and avoid using the Realtime priority altogether. Setting a processor-bound process to Realtime could cause the computer to stop responding altogether. Decreasing the priority of a process might prevent it from running at all, not merely force it to run less frequently. In addition, lowering priority does not necessarily reduce the amount of processor time a thread receives; this happens only if it is no longer the highest-priority thread.
11.11.1 Process affinity On symmetric multi-processing (SMP) systems, the Windows scheduler distributes the load of ready threads over all available processors based on thread priority. Even though Windows will often try to associate known threads with a specific CPU (called soft affinity), threads invariably end up distributed among multiple processors.
330
Tuning IBM System x Servers for Performance
Hard affinity can be applied to permanently bind a process to a given CPU or set of CPUs, forcing the designated process to always return to the same processor. The performance advantage in doing this is best seen in systems with large Level 2 caches as the cache hit ratio will improve dramatically. Assigning hard processor affinity to assign processes to CPUs is not typically used as a method for improving system performance. The only circumstances under which it will occasionally be employed are those servers where multiple instances of the same application are running on the same system, such as SQL Server or Oracle. As with all tuning techniques, the performance of the system should be measured to determine whether using process affinity has actually offered any tangible benefit. Determining the correct mix of processes assigned to the CPUs in the system can be time-consuming. Some applications, like SQL Server, provide internal options to assign themselves to specific CPUs. The other method for setting affinity is through Task Manager: 1. Right-click the process in question 2. Click Set Affinity as shown in Figure 11-14. 3. Select the CPUs to which you want to restrict the process and click OK.
Task Manager allows you to assign processor affinity to selected processes.
Figure 11-14 Assigning Processor Affinity to a selected process
Chapter 11. Microsoft Windows Server
331
Note that like changing the process’s priority, changing process affinity in this manner will only last for the duration of the process. If the process ends or the system is rebooted, the affinity will need to be reallocated as required. Note also that not all processes permit affinity changes. Note too that the Set Affinity option described above in the Task Manager application will only display in the context menu on a system with multiple logical or physical processors, including processors with multiple cores.
11.12 Assigning interrupt affinity Microsoft offers a utility called Intfiltr that allows the binding of device interrupts to specific system processors. This partitioning technique can be employed to improve system performance, scaling and the partitioning of large servers. Specifically for server performance tuning purposes, Intfiltr allows you to assign the interrupts generated by each network adapter to a specific CPU. Of course, it is only useful on SMP systems with more than one network adapter installed. Binding the individual network adapters in a server to a given CPU can offer large performance efficiencies. Intfiltr uses plug-and-play features of Windows that permits affinity for device interrupts to particular processors. Intfiltr binds a filter driver to devices with interrupts and is then used to set the affinity mask for the devices that have the filter driver associated with them. This permits Windows to have specific device interrupts associated with nominated processors.
332
Tuning IBM System x Servers for Performance
Figure 11-15 Assigning processor affinity using the INTFILTR tool
Interrupt filtering can affect the overall performance of your computer, in both a positive and negative manner. Under normal circumstances, there is no easy way to determine which processor is best left to handle specific interrupts. Experimentation and analysis will be required to determine whether interrupt affinity has yielded performance gains. To this end, by default, without tools like Intfiltr, Windows directs interrupts to any available processor. Note some consideration needs to be made when configuring Intfiltr on a server with CPUs that supports Hyper-Threading to ensure that the interrupts are assigned to the correct physical processors desired, not the logical processors. Assigning interrupt affinity to two logical processors that actually refer to the same physical processor will obviously offer no benefit and can even detract from system performance. Interrupt affinity for network cards can offer definite performance advantages on large, busy servers with many CPUs. Our recommendation is to trial Intfiltr in a test environment to associate specific interrupts for network cards with selected processors. Note that this test environment should simulate as close to your production environment as possible, including hardware, operating system and application configuration. This will allow you to determine if using interrupt affinity is going to offer a performance advantage for your network interfaces.
Chapter 11. Microsoft Windows Server
333
Note: You can use Intfiltr to create an affinity between CPUs and devices other than network cards also, such as disk controllers. Experimentation again is the best way to determine potential performance gains. To determine the interrupts of network cards or other devices, use Windows Device Manager or, alternately, run System Information (WINMSD.EXE). The Intfiltr utility and documentation is available free of charge from Microsoft: ftp://ftp.microsoft.com/bussys/winnt/winnt-public/tools/affinity/intfiltr.zip For more information, see: http://support.microsoft.com/?kbid=252867
11.13 The /3GB BOOT.INI parameter (32-bit x86) By default, the 32-bit (x86) editions of Windows can address a total of 4 GB of virtual address space. This is a constraint of the 32-bit (x86) architecture. Normally, 2 GB of this is reserved for the operating system kernel requirements (privileged-mode) and the other 2 GB is reserved for application (user-mode) requirements. Under normal circumstances, this creates a 2 GB per-process address limitation. Windows provides a /3GB parameter to be added to the BOOT.INI file that reallocates 3 GB of memory to be available for user-mode applications and reduces the amount of memory for the system kernel to 1 GB. Some applications, such as Microsoft Exchange and Microsoft SQL Server, written to do so, can derive performance benefits from having large amounts of addressable memory available to individual user-mode processes. In such instances, having as much free space for user-mode processes is desirable. Given the radically increased memory capability of 64-bit (x64) operating systems, the /3GB switch nor the /PAE switch (described in 11.14, “Using PAE and AWE to access memory above 4 GB (32-bit x86)” on page 335) are not for use on the 64-bit (x64) editions of the Windows Server 2003 operating system. To edit the BOOT.INI file to make this change, complete the following steps: 1. Open the System Control Panel. 2. Select the Advanced tab. 3. Within the Startup and Recovery frame, click Settings. 4. Click Edit. Notepad opens, and you can edit the current BOOT.INI file.
334
Tuning IBM System x Servers for Performance
5. Edit the current ARC path to include the /3GB switch, as shown in Figure 11-16. 6. Restart the server for the change to take effect.
Add the /3GB switch to enable more addressable memory for user-mode applications
Figure 11-16 Editing the BOOT.INI to include the /3GB switch
You normally use this switch only when a specific application recommends its use. Typically, you use it where applications have been compiled to use more than 2 GB per process, such as some components of Exchange. For more information, see: http://support.microsoft.com/kb/291988 http://support.microsoft.com/kb/851372 http://support.microsoft.com/kb/823440
11.14 Using PAE and AWE to access memory above 4 GB (32-bit x86) As described in 11.13, “The /3GB BOOT.INI parameter (32-bit x86)” on page 334, the native 32-bit architecture of the x86 processor allows a maximum addressable memory space of 4 GB. The Intel Physical Address Extension (PAE) is a 36-bit memory addressing mode that allows 32-bit (x86) systems to address memory above 4 GB. PAE requires appropriate hardware and operating system support to be implemented. Intel introduced PAE 36-bit physical addressing with the Intel Pentium Pro processor. Windows has supported PAE since Windows NT Server 4.0, Enterprise Edition and is supported with the Advanced and Datacenter Editions of Windows 2000 Server and the Enterprise and Datacenter Editions of Windows Server 2003.
Chapter 11. Microsoft Windows Server
335
Windows uses 4 KB pages with PAE to map up to 64 GB of physical memory into a 32-bit (4 GB) virtual address space. The kernel effectively creates a “map” in the privileged mode addressable memory space to manage the physical memory above 4 GB. The 32-bit (x86) editions of Windows Server 2003 allow for PAE through use of a /PAE switch in the BOOT.INI file. This effectively allows the operating system to use physical memory above 4 GB. As the 64-bit (x64) editions of Windows are not bound by this same memory architecture constraint, the PAE switch is not used in these versions of the Windows Server 2003 operating system. Even with PAE enabled, the underlying architecture of the system is still based on 32-bit linear addresses. This effectively retains the usual 2 GB of application space per user-mode process and the 2 GB of kernel mode space because only 4 GB of addresses are available. However, multiple processes can immediately benefit from the increased amount of addressable memory because they are less likely to encounter physical memory restrictions and begin paging. Address Windowing Extensions (AWE) is a set of Windows APIs that take advantage of the PAE functionality of the underlying operating system and allow applications to directly address physical memory above 4 GB. Some applications like SQL Server 2000, Enterprise Edition, have been written with these APIs and can harness the significant performance advantages of being able to address more than 2 GB of memory per process. More recent applications, like SQL Server 2005 x64 Editions, have been written to take advantage of the 64-bit (x64) memory architecture and can deliver enormous performance gains over their 32-bit (x86) counterparts. They do not need to use AWE to address memory above 4 GB. To edit the BOOT.INI file to enable PAE, complete the following steps: 1. Open the System Control Panel. 2. Select the Advanced tab. 3. Within the Startup and Recovery frame, click Settings. 4. Click Edit. Notepad opens, and you can edit the current BOOT.INI file. 5. Edit the current ARC path to include the /PAE switch as shown in Figure 11-17. 6. Restart the server for the change to take effect.
336
Tuning IBM System x Servers for Performance
Figure 11-17 Editing the BOOT.INI to include the /PAE switch
For more information, see: http://support.microsoft.com/kb/283037 http://support.microsoft.com/kb/268363 http://support.microsoft.com/kb/823440
11.14.1 Interaction of the /3GB and /PAE switches There is often confusion between when to use the /3GB switch and when to use the /PAE switch in the BOOT.INI file. In some cases it is desirable to use both. Recall the following information previously covered: The /3GB switch reallocates the maximum 4 GB addressable memory from the normal 2 GB for user-mode applications and 2 GB for the kernel to allow 3 GB of physical memory to be used for applications, leaving 1 GB for the system kernel. PAE permits the operating system to see and make use of physical memory above and beyond 4 GB. This is achieved through the use of the 1 or 2 GB of kernel addressable memory (depending on the use of the /3GB switch) to “map” and manage the physical memory above 4 GB. Applications written using AWE make use of PAE to allow individual applications (processes) to use more than the 2 GB limitation per process. On a server with between 4 GB and 16 GB of RAM hosting applications that have been compiled or written with AWE to use more than 2 GB of RAM per process or hosting many applications (processes), each contending for limited physical memory, it would be desirable to use both the /3GB and /PAE switches. This will deliver the best performance possible for such a system. Servers with more than 16 GB of physical memory should not use both the /3GB switch and the /PAE switch. The /PAE switch is obviously required to make use of all physical memory above 4 GB. Remember however that PAE uses the kernel addressable memory to manage the physical memory above 4 GB. When physical memory exceeds 16 GB, the 1 GB of memory allocated to the kernel
Chapter 11. Microsoft Windows Server
337
when the /3GB switch is used is not sufficient to manage all the additional physical memory above 4 GB. Thus, only the /PAE switch should be used in such a case to avoid the system running out of kernel memory. For more information, see: http://msdn2.microsoft.com/en-us/library/ms175581.aspx
11.15 TCP/IP registry optimizations Windows has several registry parameters that can be modified to optimize the performance of the TCP/IP protocol for your environment. In most networks, Windows TCP/IP is tuned by the operating system to achieve maximum performance, however there might be settings that can be made to improve performance on your network. When changing these parameters, you should understand what the parameters are doing and what will be the ramification of changing them. If you find that any of these changes cause throughput of your server to decrease, then you should reverse the changes made. Tip: Several of the registry values described in this chapter do not exist in the registry by default and must be added manually. For these values, there is a system default which is overridden when you create the registry value. This section and the next list a series of modifications that can, however, be made to the system registry that under certain circumstances might offer system performance improvements. Almost all of these modifications are ones for which there is no corresponding control panel or other GUI utility built natively into Windows to allow these values to be tuned without using the registry directly. Modifying the registry is not for the faint-hearted and should only be done by experienced technical people. Incorrectly modifying the registry can cause serious system instability and in some circumstances, stop a server from booting correctly. Before implementing any of the suggestions made here, ensure you completely understand their impact and downstream effect they might have. Further reading from the Microsoft Web site or elsewhere is recommended to gain more understanding of the parameter changes listed in this section. Where useful further references exist, these have been provided. Note that several of these changes might not take effect until the server is rebooted subsequent to making the change. Note too that when several of these changes are made and hard-coded into the registry, they restrict Windows from self-tuning this value to what might, under different circumstances, be a more
338
Tuning IBM System x Servers for Performance
optimal setting. Thus, any modifications made should only be to affect the server performance for the role it is currently hosting. Note: Tuning Windows TCP/IP settings can have a significant impact on memory resources. Monitoring memory usage is very important if you choose to implement any of the settings suggested this section.
11.15.1 TCP window size The TCP receive window specifies the maximum number of bytes that a sender can transmit without receiving an acknowledgment from the receiver. The larger the window size, the fewer acknowledgements are sent back, and the more optimal the network communications are between the sender and receiver. Having a smaller window size reduces the possibility that the sender will time-out while waiting for an acknowledgement, but will increase network traffic and reduce throughput. TCP dynamically adjusts to a whole multiple of the maximum segment size (MSS) between the sender and receiver. The MSS is negotiated when the connection is initially set up. By adjusting the receive window to a whole multiple of the MSS, the number of full-sized TCP segments used during data transmission is increased, improving throughput. By default, TCP will try to automatically negotiate the optimal window size depending on the maximum segment size. It initially starts at 16 KB and can range to a maximum of 64 KB. The TCP window size can also be statically specified in the registry, permitting potentially larger or more efficient values than can be negotiated dynamically. The maximum TCP window size is normally 65535 bytes (64 KB). The maximum segment size for Ethernet networks is 1460 bytes. The maximum multiple of increments of 1460 that can be reached before exceeding this 64 KB threshold is 62420 bytes. This value of 62420 can thus be set in the registry for optimal performance on high-bandwidth networks. The value does not ordinarily exist in the registry and must be added. The TcpWindowSize registry value can be set at a global or per-interface level. Interface-level settings will override the global value. To achieve the maximum window size, we recommended to set this only at the global level.
Chapter 11. Microsoft Windows Server
339
The registry value recommendation is as follows: Key: Value: Data type: Range: Default: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters TCPWindowSize REG_DWORD 0x0 to 0xFFFF 0x4470 (17520 bytes, the Ethernet MSS (1470)) multiple closest to 16 K) 0xFAF0 (62420) No, needs to be added.
For more information, see: http://support.microsoft.com/?kbid=263088 http://support.microsoft.com/?kbid=224829
11.15.2 Large TCP window scaling and RTT estimation (timestamps) The following TCP features are described in RFC 1323. For more efficient use of high bandwidth networks, an even larger TCP window size can be used than described above in 11.15.1, “TCP window size” on page 339. This feature is new to Windows 2000 and Windows Server 2003 and is referred to as TCP window scaling and is used to increase the maximum TCP window size from the previous limit of 65535 bytes (64 KB) up to 1073741824 bytes (1 GB). With large window (scaling window) support enabled, Windows can dynamically recalculate and scale the window size. This enables more data to be transmitted between acknowledgements, increasing throughput and performance. The amount of time used for round-trip communications between a sender and receiver is referred to by TCP as the round-trip time (RTT). A time stamp option available to TCP improves the accuracy of RTT values by calculating it more frequently. This option is particularly helpful in estimating RTT over longer round-trip WAN links and will more accurately adjust TCP retransmission time-outs. This time stamp option provides two time stamp fields in the TCP header, one to record the initial transmission time and the other to record the time on the receiver. The time stamp option is particularly valuable when window scaling support is enabled to ensures the integrity of the much larger packets being transmitted without acknowledgement. Enabling timestamps might actually have a slight impact to throughput as it adds 12 bytes to the header of each packet. The value of data integrity versus maximum throughput will thus need to be evaluated. In
340
Tuning IBM System x Servers for Performance
some instances, such as video streaming, where large TCP window size might be advantageous, data integrity might be secondary to maximum throughput. In such an instance, window scaling support can be enabled without timestamps. Support for scaling window size and timestamps is negotiated at connection setup between the sender and receiver. Only if both the sender and receiver have support for these features enabled, will they be used during data transmission. A small TCP window size will be negotiated initially and over time, using internal algorithms, the window size will grow to the maximum specified size. To enable support for scaling Windows or improved time stamps (RTT) estimation, make the following changes to the registry. Key: Value: Data type: Range: Default: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters TCP1323Opts REG_DWORD 0x0 - 0x3 (see Table 11-6) 0x0 0x3 No, needs to be added.
For more information, see: http://support.microsoft.com/?kbid=224829 Note: The registry entry for this value is a 2-bit bitmask. The lower-bit determines whether scaling is enabled, and the higher-bit determines whether timestamps are enabled. Table 11-6 Possible entries for TCP1323Opts registry value
TCP1323Opts registry value
Result
0x0
Disable windows scale and timestamps
0x1
Windows scaling enabled only
0x2
Timestamps scaling enabled only
0x3 (recommended)
Windows scaling and timestamps enabled
After TCP1323Opts has been used to enabled TCP window scaling, the registry value TCPWindowSize described in 11.15.1, “TCP window size” on page 339 can be increased to value from 64K (65535 bytes) right through to 1 GB (1,073,741,824 bytes). For best performance and throughput, the value set here should be a multiple of the maximum segment size (MSS).
Chapter 11. Microsoft Windows Server
341
As the optimal value for TCPWindowSize with window scaling support enabled will be different for each implementation, no specific recommendation is made here. This should be determined through careful testing in your environment. Note that as the window size increases, so does the risk of data corruption and subsequent resends as fewer acknowledgements will be issued from the receiver. This might actually then have a negative impact on performance.
11.15.3 TCP connection retransmissions The number of times that TCP will retransmit an unacknowledged connection request (SYN) before aborting is determined by the registry value TcpMaxConnectRetransmissions. For each given attempt, the retransmission time-out is doubled with each successive retransmission. For Windows Server 2003, the default number of time-outs is 2 and the default time-out period is 3 seconds (set by the TCPInitialRTT registry entry). The parameter for connections retransmissions can be incremented to prevent a connection from timing out across slow WAN links. As the optimal value will be different for each implementation, no specific recommendation is made. This should be determined through careful testing in your environment. Note: this parameter should not be set so high that the connection will not time out at all. Key: Value: Data type: Range: Default: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters TCPMaxConnectRetransmissions REG_DWORD 0x0 - 0xFF 0x2 None made - environment specific No, needs to be added.
For more information, see: http://support.microsoft.com/?kbid=120642
11.15.4 TCP data retransmissions The number of times that TCP will retransmit an unacknowledged data segment before aborting is specified by the registry value TcpMaxDataRetransmissions. The default value is 5 times. TCP establishes an initial interval by measuring the round trip for a given connection. With each successive retransmission attempt, the interval doubles until responses resume or time-out altogether - at which time the interval is reset to the initially calculated value.
342
Tuning IBM System x Servers for Performance
As the optimal value will be different for each implementation, no specific recommendation is made here. This should be determined through careful testing in your environment. Key: Value: Data type: Range: Default: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters TCPMaxDataRetransmissions REG_DWORD 0x0 - 0xFFFFFFFF 0x5 None made, environment specific No, needs to be added.
For more information, see: http://support.microsoft.com/?kbid=120642
11.15.5 TCP TIME-WAIT delay By default, TCP will normally allocate a port with a value between 1024 and 5000 for a socket request for any available short-lived (ephemeral) user port. When communications over a given socket have been closed by TCP, it waits for a given time before releasing it. This is known as the TIME-WAIT delay. The default setting for Windows Server 2003 is two minutes, which is appropriate for most situations. However, some busy systems that perform many connections in a short time might exhaust all ports available, reducing throughput. Windows has two registry settings that can be used to control this time-wait delay: TCPTimedWaitDelay adjusts the amount of time that TCP waits before completely releasing a socket connection for re-use. MaxUserPort sets the number of actual ports that are available for connections, by setting the highest port value available for use by TCP. Reducing TCPTimedWaitDelay and increasing MaxUserPort can increase throughput for your system. Note: These changes only optimize performance on exceptionally busy servers hosting thousands of simultaneous TCP connections, such as a heavily loaded LDAP, FTP or Web servers.
Chapter 11. Microsoft Windows Server
343
Key: Value: Data type: Range: Default: Recommendation: Value exists by default: Key: Value: Data type: Range: Default: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters TCPTimedWaitDelay REG_DWORD 0x0 - 0x12C (0 - 300 seconds) 0x78 (120 seconds) 0x1E (30 seconds) No, needs to be added. HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters MaxUserPort REG_DWORD 0x1388 - 0xFFFE (5000 - 65534) 0x1388 (5000) 0xFFFE No, needs to be added.
Note: The value name is MaxUserPort, not MaxUserPorts. For more information, see Microsoft Windows Server 2003 TCP/IP Implementation Details, which is available from: http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technolo gies/networking/tcpip03.mspx
11.15.6 TCP Control Block (TCB) table For each active TCP connection on a system, various control variables about the connection are stored in a memory block called a TCP control block (TCB). These TCB values are initialized when the connection is established and then continually updated throughout the lifetime of the connection. Each of these TCBs are maintained in a hash table called the TCB table. The size of the TCB table is controlled by the registry value MaxHashTableSize. On a large system with many active connections, having a larger table reduces the amount of time spent the system must spend to locate a particular TCB. By partitioning the TCB table, contention for table access is minimized. TCP performance can be optimized by increasing the number of partitions. This is particularly so on multi-processor systems. The registry value NumTcbTablePartitions value controls the number of partitions. By default, the value is the square of the number of processors in the system.
344
Tuning IBM System x Servers for Performance
The MaxHashTableSize value should always be of a power of two to correctly function. You might consider using the maximum value for large servers that might host a high number of connections. Bear in mind however that the table uses non-paged pool memory so do not set too high a value if the system is constrained on non-paged pool memory of course if the system simply will not support a high load of connections. With a server that has more than one CPU, the NumTcbTablePartitions parameter should be four times the number of processors installed in the system In most cases this will perform equally or better than the default square of the number of CPUs in the system, especially on servers with 8 or 16 CPUs, where too high a value of NumTcbTablePartitions can impact CPU performance. Key: Value: Data type: Range: Default: Recommendation: Value exists by default: Key: Value: Data type: Range: Recommendation: Default:
HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters MaxHashTableSize REG_DWORD 0x40 - 0x10000 (1 - 65536), should be a power of 2 (2n) 0x200 (512) 0x10000 (65536) No, needs to be added. HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters NumTcbTablePartitions REG_DWORD 0x1 - 0xFFFF (1 - 65535), should be a power of 2 (2n) 4 x the number of processors in the system n2, where n is the number of CPUs installed
Tip: Because this value does not exist in the registry by default, be careful to ensure the value set is NumTcbTablePartitions, not NumTcpTablePartitions. For more information, see Microsoft Windows Server 2003 TCP/IP Implementation Details, which is available from: http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technolo gies/networking/tcpip03.mspx TCBs are normally pre-allocated in memory to avoid spending time allocating and de-allocating TCBs every time TCP connections are established and closed. The reuse or caching of TCBs improves memory management but also restricts how many active connections TCP can support at a given time. The registry value MaxFreeTcbs configures the threshold number of connections required before TCBs in the TIME-WAIT state (that is, the connection has been
Chapter 11. Microsoft Windows Server
345
closed but is not free for reuse yet), are re-used. This value was often set higher than the default to optimize performance in older Windows NT implementations to ensure there was always sufficient pre-allocated TCBs. Since Windows 2000, a feature was added to decrease the chance of running out of pre-allocated TCBs. If more than the number of TCBs specified in the new registry value MaxFreeTWTcbs are in the TIME-WAIT state then all connections that have been in the TIME-WAIT state for longer than 60 seconds are forcibly closed and made available for use again. With this feature now incorporated into Windows 2000 Server and now Windows Server 2003, modification of the MaxFreeTcbs registry value is no longer deemed valuable in optimizing TCP performance.
11.15.7 TCP acknowledgement frequency TCP uses delayed acknowledgements to reduce the number of packets transmitted in the network, thereby improving performance. Normally, an acknowledgement (ACK) is sent every second TCP segment. This value can be increased to reduce the cost of processing of network traffic, especially in the case of large data uploads from the client to the server. Note: This value is configured at the interface level for each network interface installed in the system and differs depending on the speed of the interface. The default value is 2. For Fast Ethernet (100 Mbps) network interfaces, use a value of 5 (0x5). For Gigabit (1000 Mbps) network interfaces, use a value of 13 (0xD). Key:
Value: Data type: Range: Default: Recommendation: Value exists by default:
HKLM\System \CurrentControlSet \Services\Tcpip\Parameters\Interface\xx (xx depends on network interface) TcpAckFrequency REG_DWORD 0x1 - 0xD (1-13) 2 0x5 (5) for FastEthernet or 0xD (13) for Gigabit interfaces No, needs to be added.
For more information, see Performance Tuning Guidelines for Windows Server 2003, which is available from: http://www.microsoft.com/windowsserver2003/evaluation/performance/tuning.mspx
346
Tuning IBM System x Servers for Performance
11.15.8 Maximum transmission unit The TCP/IP maximum transmission unit (MTU) defines the maximum size of an IP datagram that can be transferred in one frame over the network over a specific data link connection. The MTU might differ for network segments between the sender and receiver. Too small a packet size means data transmission is inefficient however too large a packet size means that data might exceed the MTU of the links over which the packet is transmitted. One method of determining the most efficient packet size allows routers to divide packets as they encounter a network segment with a smaller MTU than that of the packet being transmitted. This is referred to as IP segmentation or fragmentation. This method places extra overhead on routers in need to divide and reassemble packets. A preferred and more common option is for a client to determine the maximum MTU that can be used on all segments between the sender and receiver. The client communicates with routers along the path as required to determine the smallest MTU that can be used in all segments from start to end. This process is known as calculating the path maximum transmission unit (PMTU) and will result in the most efficient packet size that will not need to be fragmented. TCP/IP normally determines the optimal MTU dynamically, as described in 11.15.9, “Path Maximum Transmission Unit (PMTU) Discovery” on page 349. Windows does however allow the MTU to be statically configured. This is not normally recommended but might be suitable in some environments. Under most circumstances, allowing TCP/IP to dynamically determine the MTU is preferred, unless you can be certain of the MTU for every segment in your environment that a host might need to communicate over. By setting it statically, the MTU will not need to be negotiated and can offer a a performance improvement. The MTU for your environment can be determined by using the PING command on one of your servers and issuing the following command: PING -f -l <MTUsize> Tip: The f and l parameters must be in lower-case for this command to work. The MTUSize parameter is one you will use to determine the PMTU between the server and a remote host that it communicates with frequently. This might be a host on the same segment or one across multiple inter-continental WAN links. The command should be issued repeatedly using different values for MTUSize until the highest possible PMTU setting is achieved without receiving a “packet needs to be fragmented” response.
Chapter 11. Microsoft Windows Server
347
A good value to start with for MTUSize is 1500, the Windows default. Work up or down from there (normally down) until the maximum setting is determined. Example 11-1 shows an optimal setting of a 1472 byte MTU before the packet starts being fragmented at 1473 bytes. Example 11-1 Determining the MTU size
C:\>ping -f -l 1472 w3.ibm.com Pinging w3ibm.southbury.ibm.com [9.45.72.138] with 1472 bytes of data: Reply Reply Reply Reply
from from from from
9.45.72.138: 9.45.72.138: 9.45.72.138: 9.45.72.138:
bytes=1472 bytes=1472 bytes=1472 bytes=1472
time=26ms time=26ms time=26ms time=26ms
TTL=245 TTL=245 TTL=245 TTL=245
Ping statistics for 9.45.72.138: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 26ms, Maximum = 26ms, Average = 26ms C:\>ping -f -l 1473 w3.ibm.com Pinging w3ibm.southbury.ibm.com [9.45.72.138] with 1473 bytes of data: Packet Packet Packet Packet
needs needs needs needs
to to to to
be be be be
fragmented fragmented fragmented fragmented
but but but but
DF DF DF DF
set. set. set. set.
Ping statistics for 9.45.72.138: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), Tip: This registry value is set at the interface level, not as an overall TCP/IP parameter.
348
Tuning IBM System x Servers for Performance
After you determine this optimal MTU value, you set it in the registry as follows: Key:
Value: Data type: Range: Default: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters\Interface\xxxxxxx (depends on network interface) MTU REG_DWORD 0x44 (68), determined dynamically MTU OR 0xFFFFFFFF 0xFFFFFFFF (determine dynamically PMTU) 0xFFFFFFFF No, needs to be added.
For more information, see http://www.microsoft.com/windows2000/techinfo/reskit/en-us/regentry/58792.asp Important: There is a close interaction between the MTU registry setting and the registry setting described in the next section, EnablePMTUDiscovery. To use a statically determined value for MTU as described above, EnablePMTUDiscovery should be disabled (that is, set to 0). If EnablePMTUDiscovery is disabled and no value is set for MTU as described above, TCP/IP will configure a default MTU size of 576 bytes. This packet size usually avoids any packet fragmentation but is far from optimal for most environments. This means that in most instances, if EnablePMTUDiscovery is disabled, a value should be set for MTU as well. If EnablePMTUDiscovery is enabled (the default), then the MTU registry value should be set to 0xFFFFFFFF, or it can be removed altogether.
11.15.9 Path Maximum Transmission Unit (PMTU) Discovery Under most circumstances, the maximum transmission unit (MTU) of every network segment that a server might possibly communicate over will not be known. Remote networks will often have an entirely different MTU to that of local networks. When enabled, the registry value EnablePMTUDiscovery lets TCP/IP automatically determine the MTU for all networks along the path to a remote host. When the MTU has been determined for all network segments on a path, it will use the highest, and thus most efficient MTU value that can used without packet fragmentation occurring. PMTU detection is enabled by default in Windows. As the EnablePMTUDiscovery does not ordinarily exist in the registry, it would normally only be created with the
Chapter 11. Microsoft Windows Server
349
intention of disabling PMTU detection. If you do choose to disable PMTU detection, a default MTU of 576 bytes will be used for all communications, unless a value is also set for the MTU registry value as described in 11.15.8, “Maximum transmission unit” on page 347. Note: This registry value applies to all network interfaces. The following registry value controls the use of PMTU Discovery: Key: Value: Data type: Range: Default: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters EnablePMTUDiscovery REG_DWORD 0 or 1 1 1 No, needs to be added.
For more information, see http://www.microsoft.com/windows2000/techinfo/reskit/en-us/regentry/58752.asp
11.16 Memory registry optimizations For most purposes, Windows operates very well in its native self-tuning capacity. Nonetheless there are many other registry changes relating to the memory subsystem that can be modified to improve system performance under specific circumstances. Some of the ones that have been noted to improve performance in production environments are listed here. Note that several of the memory specific tuning parameters listed here hold relevance only for the 32-bit (x86) versions of the Windows Server 2003 operating system. They are no longer valid for the 64-bit (x64) editions given the greatly expanded memory architecture. As with all changes, ensure you have a working and tested backup of the registry and entire server before making the change. Changes should be made and tested only one at a time. If system performance is negatively affected by making such a change, it should be reversed immediately.
11.16.1 Disable kernel paging Servers with sufficient physical memory might benefit from disabling portions of the Windows operating system kernel and user-mode and kernel-mode drivers
350
Tuning IBM System x Servers for Performance
from being paged to disk. This registry setting forces Windows to keep all components of the kernel (or executive) and drivers in memory and, thus, allows much faster access to them when required. Key: Value: Data type: Range: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Control \Session Manager\Memory Management DisablePagingExecutive REG_DWORD 0x0 (default) or 0x1 0x1 No, needs to be added
For more information, see: http://support.microsoft.com/?kbid=184419
11.16.2 Optimizing the Paged Pool Size (32-bit x86) Windows allocates memory in pools for the operating system and its components, which processes access through the use of kernel mode. Two pools of kernel mode memory exist: The paged pool (which can be paged to the pagefile) The non-paged pool (which can never be paged) Performance and system stability can be seriously impacted if Windows experiences memory resource constraints and is unable to assign memory to these pools. The amount of physical memory assigned to these two pools is assigned dynamically at system boot time. The maximum default size on 32-bit (x86) editions of Windows Server 2003 for the paged memory pool is 491 MB, and 256 MB for the non-paged pool. In the 64-bit (x64) editions of Windows Server 2003, both the paged pool and non-paged pool have a limit of 128 GB. As a result the following values do not apply for the 64-bit (x64) editions.
Chapter 11. Microsoft Windows Server
351
Some applications and workloads can demand more pooled memory than the system has been allocated by default. Setting the PagedPoolSize registry value as listed in Table 11-7 can assist in ensuring sufficient pooled memory is available. Changing this setting requires a restart of the operating system. Table 11-7 PagedPoolSize values - 32-bit (x86) Editions of Windows Server 2003
PagedPoolSize value
Meaning
0x0 (default)
Requests that the system will dynamically calculate an optimal value at system startup for the paged pool based on the amount of physical memory in the computer. This value will change if more memory is installed in the computer. The system typically sets the size of the paged pool to approximately twice that of the nonpaged pool size.
Range: 0x1 0x20000000 (512 MB)
Creates a paged pool of the specified size, in bytes. This takes precedence over the value that the system calculates, and it prevents the system from adjusting this value dynamically. Limiting the size of the paged pool to 192 MB (or smaller) lets the system expand the file system (or system pages) virtual address space up to 960 MB. This setting is intended for file servers and other systems that require an expanded file system address space (meaning slightly faster access) at the expense of being able to actually cache less data. This only makes sense if you know the files your server frequently accesses already fit easily into the cache.
0xFFFFFFFF
With this value, Windows will calculate the maximum paged pool allowed for the system. For 32-bit systems, this is 491 MB. This setting is typically used for servers that are attempting to cache a very large number of frequently used small files, some number of very large size files, or both. In these cases, the file cache that relies on the paged pool to manage its caching it able to cache more files (and for longer periods of time) if more paged pool is available.
Setting this value to 0xB71B000 (192 MB) provides the system with a large virtual address space, expandable to up to 960 MB. Note that a corresponding entry of zero (0) is required in the SystemPages registry value for this to take optimal effect.
352
Tuning IBM System x Servers for Performance
Key: Value: Data type: Range: Recommendation:
Value exists by default: Key: Value: Data type: Range: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Control \Session Manager\Memory Management PagedPoolSize REG_DWORD 0x0 (default) to 0xFFFFFFFF (4294967295) 0xB71B000 (192000000) for native SMB shares. For NFS shares when using Services for UNIX, use a value of 0xFFFFFFFF (4294967295) Yes HKLM\SYSTEM \CurrentControlSet \Control \Session Manager\Memory Management SystemPages REG_DWORD 0x0 (default) to 0xFFFFFFFF 0x0 Yes
For more information, see: http://www.microsoft.com/resources/documentation/windows/2000/server/re skit/en-us/core/fnec_evl_fhcj.asp
11.16.3 Increase memory available for I/O locking operations By default, the 32-bit (x86) editions of Windows Server 2003 sets a limit to the amount of memory that can be set aside for I/O locking operations at 512 KB. Depending on the amount of physical memory in a server, this can be increased in various increments as per Table 11-8. You can insert the values listed in this table into the registry to increase the amount of memory available for locking operations on 32-bit (x86) systems. These values hold no validity for the 64-bit (x64) editions of Windows Server 2003. Table 11-8 Maximum I/O lock limit values
Amount of physical RAM
Maximum lock limit (IoPageLockLimit value)
Less than 64 MB
Physical memory minus 7 MB
64 MB - 512 MB
Physical memory minus 16 MB
512 MB upwards
Physical memory minus 64 MB
Chapter 11. Microsoft Windows Server
353
These value ranges listed in Table 11-8 equate to those calculated in Table 11-9, depending on the exact amount of physical RAM in the machine. As almost all servers today have more than 512 MB RAM, the calculations in Table 11-9 take into account only 512 MB RAM and above. The appropriate value should be determined from the Table 11-8 and then entered into the registry value IoPageLockLimit. This value then takes precedence over the system default of 512 KB and specifies the maximum number of bytes that can be locked for I/O operations: Key: Value: Data type: Range: Recommendation: Default: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Control \Session Manager\Memory Management IoPageLockLimit REG_DWORD 0 (default) to 0xFFFFFFFF (in bytes, do not exceed this maximum!) depends on RAM, see Table 11-9 0x80000 (512 KB) No, needs to be added.
Table 11-9 Recommended settings for IoPageLockLimit
Amount of physical RAM
IoPageLockLimit Setting (hex)
512 MB
0x1C000000
1 GB (1024 MB)
0x3C000000
2 GB (2048 MB)
0x80000000
4 GB (4096 MB)
0xFC000000
8 GB (8096 MB) and above
0xFFFFFFFF
For more information, see: http://www.microsoft.com/windows2000/techinfo/reskit/en-us/regentry/29932.asp
11.16.4 Increasing available worker threads At system startup, Windows creates several server threads that operate as part of the System process. These are called system worker threads. They exist with the sole purpose of performing work on the behalf of other threads generated by the kernel, system device drivers, the system executive and other components. When one of these components puts a work item in a queue, a thread is assigned to process it.
354
Tuning IBM System x Servers for Performance
The number of system worker threads would ideally be high enough to accept work tasks as soon as they become assigned. The trade off of course is that worker threads sitting idle are using system resources unnecessarily. The DefaultNumberofWorkerThreads value sets the default number of worker threads created for a given work queue. Having too many threads needlessly consumes system resources. Having too few slows the rate at which work items are serviced. Key: Value: Data type: Range: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Services\RpcXdr\Parameters DefaultNumberofWorkerThreads REG_DWORD 0x0 (default) to 0x40 (64) 16 times the number of CPUs in the system No, needs to be added
Delayed worker threads process work items that are not considered time-critical and can have their memory stack paged out while waiting for work items. The value for AdditionalDelayedWorkerThreads increases the number of delayed worker threads created for the specified work queue. An insufficient number of threads slows the rate at which work items are serviced; a value too high will consume system resources unnecessarily. Key: Value: Data type: Range: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Control \Session Manager\Executive AdditionalDelayedWorkerThreads REG_DWORD 0x0 (default) to 0x10 (16) 0x10 (16) Yes
Critical worker threads process time-critical work items and have their stack present in physical memory at all times. The value for AdditionalCriticalWorkerThreads increases the number of critical worker threads created for a specified work queue. An insufficient number of threads slows the rate at which time-critical work items are serviced. A value that is too high consumes system resources unnecessarily. Key: Value: Data type: Range: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Control \Session Manager\Executive AdditionalCriticalWorkerThreads REG_DWORD 0x0 (default) to 0x10 (16) 0x10 (16) Yes
Chapter 11. Microsoft Windows Server
355
11.16.5 Prevent the driver verifier from running randomly The driver verifier at random intervals verifies drivers for debugging randomly. Disabling this functionality might improve system performance. Key: Value: Data type: Range: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Control \Session Manager\Memory Management DontVerifyRandomDrivers REG_DWORD 0x0 (default) or 0x1 0x1 No
For more information, see: http://www.microsoft.com/windowsserver2003/evaluation/performance/tuning.mspx
11.17 File system optimizations Several registry tuning parameters are available in Windows Server 2003. These will assist with performance in both 32-bit (x86) and 64-bit (x64) editions of the server operating system.
11.17.1 Increase work items and network control blocks The maximum number of concurrent outstanding network requests between a Windows Server Message Block (SMB) client and server is determined when a session between the client and server is negotiated. The maximum value that is negotiated is determined by registry settings on both the client and server. If these values are set too low on the server, they can restrict the number of client sessions that can be established with the server. This is particularly a problem in a Terminal Server environment where clients are typically launching many simultaneous application instances on the server itself and using many local resources. The three values that can adjusted to improve system performance for work items exist in the LanmanServer and LanmanWorkstation registry keys and are: MaxWorkItems MaxMpxCt MaxCmds Each of these values do not exist by default in the registry. The default settings for the first two values are determined by the hardware configuration of the
356
Tuning IBM System x Servers for Performance
server combined with the value of the Server dialog (File & Print Sharing setting discussed in 11.6, “File system cache” on page 309). MaxCmds has a default of 50. The MaxWorkItems value specifies the maximum number of receive buffers, or work items, that the Server service is permitted to allocate at one time. If this limit is reached, then the transport must initiate flow control, which can significantly reduce performance. The MaxMpxCt value sets the maximum number of simultaneous outstanding requests on a client to a particular server. During negotiation of the Server Message Block dialect, this value is passed to the client's redirector where the limit on outstanding requests is enforced. A higher value can increase server performance but requires more use of server work items (MaxWorkItems). The MaxCmds value specifies the maximum number of network control blocks that the redirector can reserve. The value of this entry coincides with the number of execution threads that can be outstanding simultaneously. Increase this value will improve network throughput, especially if you are running applications that perform more than 15 operations simultaneously. Take care not to set any of these values too high. The more outstanding connections that exist, the more memory resources will be used by the server. If you set the values too high, the server could run out of resources such as paged pool memory. Tip: The MaxWorkItems value must be at least four times as large as MaxMpxCt. Key: Value: Data type: Value: Default: Recommendation: Value exists by default:
HKLM\SYSTEM\CCS\Services\LanmanServer\ Parameters MaxWorkItems REG_DWORD 1 - 65535 Configured dynamically based on system resources and server setting 32768 No, needs to be added
Chapter 11. Microsoft Windows Server
357
Key: Value: Data type: Value: Default: Recommendation: Value exists by default: Key: Value: Data type: Value: Default: Recommendation: Value exists by default:
HKLM\SYSTEM\CCS\Services\ LanmanWorkstation\Parameters MaxCmds REG_DWORD 50 - 65535 50 4096 No, needs to be added HKLM\SYSTEM\CCS\Services\LanmanServer\ Parameters MaxMpxCt REG_DWORD 8192 Configured dynamically based on system resources and server setting 1 No, needs to be added
For more information, see: http://support.microsoft.com/?kbid=232476 http://support.microsoft.com/?kbid=271148
11.17.2 Disable NTFS last access updates Each file and folder on an NTFS volume includes an attribute called Last Access Time. This attribute shows when the file or folder was last accessed, such as when a user performs a folder listing, adds files to a folder, reads a file, or makes changes to a file. Maintaining this information creates a performance overhead for the file system especially in environments where a large number of files and directories are accessed quickly and in a short period of time, such as by a backup application. Apart from in highly secure environments, retaining this information might add a burden to a server that can be avoided by updating the following registry key: Key: Value: Data type: Value: Default: Recommendation: Value exists by default:
358
HKLM\SYSTEM \CurrentControlSet \Control \FileSystem NTFSDisableLastAccessUpdate REG_DWORD 0 or 1 0 1 No, needs to be added
Tuning IBM System x Servers for Performance
For more information, see: http://www.microsoft.com/resources/documentation/WindowsServ/2003/all/d eployguide/en-us/46656.asp In Windows Server 2003, this parameter can also be set by using the command: fsutil behavior set disablelastaccess 1
11.17.3 Disable short-file-name (8.3) generation By default, for every long file name created in Windows, NTFS generates a corresponding short file name in the older 8.3 DOS file name convention for compatibility with older operating systems. In many instances this functionality can be disabled, offering a performance increase. Note that before disabling short name generation, ensure that there is no DOS or 16-bit application running on the server that requires 8.3 file names or that are there any users accessing the files on the server through 16-bit applications or older file systems or operating systems. Note too that even some recent applications have been known to exhibit problems at installation and run time if this setting is made. Key: Value: Data type: Value: Default: Recommendation: Value exists by default:
HKLM\SYSTEM \CurrentControlSet \Control \FileSystem NTFSDisable8dot3NameCreation REG_DWORD 0 or 1 0 1 Yes
In Windows Server 2003, this parameter can also be set by using the command: fsutil behavior set disable8dot3 1
11.17.4 Use NTFS on all volumes Windows offers multiple file system types for formatting drives, including NTFS, FAT, and FAT32. NTFS is always be the file system of choice for servers. NTFS offers considerable performance benefits over the FAT and FAT32 file systems and should be used exclusively on Windows servers. In addition, NTFS offers many security, scalability, stability and recoverability benefits over FAT. Under previous versions of Windows, FAT and FAT32 were often implemented for smaller volumes (say <500 MB) because they were often faster in such
Chapter 11. Microsoft Windows Server
359
situations. With disk storage relatively inexpensive today and operating systems and applications pushing drive capacity to a maximum it is unlikely that such small volumes will be warranted. FAT32 scales better than FAT on larger volumes but is still not an appropriate file system for Windows servers. FAT and FAT32 have often been implemented in the past as they were seen as more easily recoverable and manageable with native DOS tools in the event of a problem with a volume. Today, with the various NTFS recoverability tools built both natively into the operating system and as third-party utilities available, there should no longer be a valid argument for not using NTFS for file systems.
11.17.5 Do not use NTFS file compression While it is an easy way to reduce space on volumes, NTFS file system compression is not appropriate for enterprise file servers. Implementing compression places an unnecessary overhead on the CPU for all disk operations and is best avoided. Think about options for adding additional disk, near-line storage or archiving data before seriously considering file system compression.
11.17.6 Monitor drive space utilization The less data a disk has on it, the faster it will operate. This is because on a well defragmented drive, data is written as close to the outer edge of the disk as possible, as this is where the disk spins the fastest and yields the best performance. Disk seek time is normally considerably longer than read or write activities. As noted above, data is initially written to the outside edge of a disk. As demand for disk storage increases and free space reduces, data is written closer to the center of the disk. Disk seek time is increased in locating the data as the head moves away from the edge, and when found, it takes longer to read, hindering disk I/O performance. This means that monitoring disk space utilization is important not just for capacity reasons but for performance also. It is not practical nor realistic to have disks with excessive free space however. Tip: As a rule of thumb, work towards a goal of keeping disk free space between 20% to 25% of total disk space. See 9.6.3, “Active data set size” on page 203 for a further discussion on this topic and the performance benefits.
360
Tuning IBM System x Servers for Performance
11.17.7 Use disk defragmentation tools regularly Over time, files become fragmented in non-contiguous clusters across disks, and system performance suffers as the disk head jumps between tracks to seek and re-assemble them when they are required. Disk defragmentation tools work to ensure all file fragments on a system are brought into contiguous areas on the disk, improving disk I/O performance. Regularly running a disk defragmentation tool on a server is a relatively easy way to yield impressive system performance improvements. Tip: We recommend you defragment your drives daily if possible. Most defragmentation tools work the fastest and achieve the best results when they have plenty of free disk space to work with. This provides another good reason to monitor and manage disk space usage on production servers. In addition, try to run defragmentation tools when the server is least busy and if possible, during scheduled system downtime. Defragmentation of the maximum number of files and directories will be achieved if carried out while applications and users that typically keep files open are not accessing the server. Windows Server 2003 includes a basic disk defragmentation tool. While offer good defragmentation features, it offers little in the way of automated running—each defragmentation process must be initiated manually or through external scripts or scheduling tools. A number of high-quality third-party disk defragmentation tools exist for the Windows operating system, including tailorable scheduling, reporting and central management functionality. The cost and performance benefit of using such tools is normally quickly realized when compared to the ongoing operational costs and degraded performance of defragmenting disks manually, or not at all.
11.17.8 Review disk controller stripe size and volume allocation units When configuring drive arrays and logical drives within your hardware drive controller, ensure you match the controller stripe size with the allocation unit size that the volumes will be formatted with within Windows. This will ensure disk read and write performance is optimal and gain better overall server performance. Having larger allocation unit (or cluster or block) sizes means the disk space is not used as efficiently, but it does ensure higher disk I/O performance as the disk head can read in more data in one read activity.
Chapter 11. Microsoft Windows Server
361
To determine the optimal setting to configure the controller and format the disks with, you should determine the average disk transfer size on the disk subsystem of a server with similar file system characteristics. Using the Windows System (Performance) Monitor tool to monitor the Logical Disk object counters of Avg. Disk Bytes/Read and Avg. Disk Bytes/Write over a period of normal activity will help determine the best value to use. Some server purposes might warrant a smaller allocation unit size where the system will be accessing many small files or records however with file sizes, file systems and disk storage every increasing today our testing has found setting an allocation unit size of 64 KB delivers sound performance and I/O throughput under most circumstances. Improvements in performance with tuned allocation unit sizes can be particularly noted when disk load increases. Note that either the FORMAT command line tool or the Disk Management tool are needed to format volumes larger than 4096 bytes (4 KB). Windows Explorer will only format up to this threshold. Note also that CHKDSK can be used to confirm the current allocation unit size of a volume however it needs to scan the entire volume before the desired information is displayed (shown as Bytes in each allocation unit). Tip: The default allocation unit size that Windows Server 2003 will format volumes with is 4096 bytes (4 KB). It is definitely worth considering whether this size really is the most appropriate for your purposes for the volumes on your servers.
11.17.9 Use auditing and encryption judiciously Auditing and file encryption are two valuable security features that while very beneficial, can also add considerable overhead to system resources and performance. In particular, CPU and memory resources can be taxed by using auditing and / or encryption. Auditing is often a required system function by security conscious organizations. The extent to which it impacts server resources will be determined by what level auditing servers are actually employed. To ensure the performance impact of auditing is kept as low as possible, ensure that only the system events, and areas of file systems and the registry that actually do require auditing are configured so. Similarly, file encryption is required in some instances to improve system security. As with auditing, ensure that only directories and files that actually require the security brought about by encryption are set up with this feature.
362
Tuning IBM System x Servers for Performance
11.18 Other performance optimization techniques This section details various other methods that should be implemented on Windows servers to extract the best performance.
11.18.1 Dedicate server roles Where budget and operational resources permit, use dedicated domain-controller servers, and dedicated member, or stand-alone, servers. Do not combine Active Directory, DNS, WINS, DHCP, Certificate Services or similar infrastructure services onto a member server that’s primary function is for another purpose. Each of these services places additional overhead on the server, taking away valuable resources that should be dedicated to the primary function the member server is serving. In the same manner, system resources are best dedicated to specific intensive functions such as file serving, line-of-business applications servers, mail servers, database servers or Web servers. The nature of each of these functions will each affect the server subsystems in a different way so to improve performance and increase stability, dedicate servers for different server functions.
11.18.2 Run system intensive operations outside peak times Applications and jobs that are intensive on any server subsystem should be scheduled to run outside peak server times. When a server needs to be running fastest, it does not make sense to slow it down with applications such as virus scanners, backup jobs or disk fragmentation utilities.
11.18.3 Log off the server console A simple but important step in maximizing your server’s performance is to keep local users logged off the console. A locally logged-on unnecessarily consumes systems resources, potentially impacting the performance of applications and services running on the system.
11.18.4 Remove CPU-intensive screen savers A server is not the place to run fancy 3D or OpenGL screen savers. Such screen-savers are known to be CPU-intensive and use important system resources when they are running. It is best to avoid installing these altogether as an option at server-build time, or to remove them if they have been installed. The basic “Windows Server 2003” or blank screen savers are the best choice.
Chapter 11. Microsoft Windows Server
363
In a similar vein, do not use memory-wasting fancy desktop wallpapers images on your server.
11.18.5 Use the latest drivers, firmware, and service packs Installing the latest version of a device driver, patch, BIOS update, microcode, or firmware revision for hardware is a very important part of routine server maintenance. Newer device drivers not only fix bugs and increase system stability, but can also increase the performance and efficiency of a device, improving overall system performance. A good example of this with System x servers is the various models of ServeRAID adapters. Revisions in the firmware and drivers for these RAID controllers has often added significant performance and functionality benefits over previous releases. Microsoft periodically issues service packs and hot fixes for their operating systems. After a period of testing in your environment, these should be deployed to production systems. Service packs and hot fixes often introduce updated code to key kernel and sub-system components of the operating system add can extra performance and functionality benefits. In addition to the performance benefits that can be offered by latest version of device drivers, patches, firmware and service packs, many hardware and software vendors will not offer support for their products until this latest revision is installed regardless. It is good maintenance practice to do so periodically.
11.18.6 Avoid the use of NET SERVER CONFIG commands As discussed, Windows is for the most part a self-tuning operating system. If a value does not ordinarily exist in the registry, creating it and manually setting it normally prevents Windows from self-tuning the value automatically. In some circumstances this is what you hope to achieve by setting values that you believe are more optimal for your environment that Windows normally auto-configures. The core Windows server service is controlled by a set of values defined in registry at: HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters In a standard clean installation, many values that can exist in this registry location do not. This allows Windows to tune these values dynamically as the operating system sees fit to optimize system performance of the server service.
364
Tuning IBM System x Servers for Performance
A command exists to set several of these parameters to desired values to avoid direct manipulation of the registry. These include the following commands: net config server /autodisconnect:time net config server /srvcomment:"text" net config server /hidden:yes|no As expected, these three commands create and manipulate the contents of the registry values autodisconnect, disc, srvcomment, and hidden in the LanmanServer\parameters key to the desired values. There is, however, a most unfortunate catch in using these commands. Not only do they create the necessary registry entry for the parameter requested, but they also create a whole series of other unrelated registry values that are extremely important to the functioning of the server service. Creating all these values sets these values to static settings and subsequently prevents them from further self-tuning by Windows. This is most undesirable and should be avoided. For example, administrators who want to hide Windows computers from the network browse list might issue the command: net config server /hidden:yes Before issuing such a command, the registry parameters for the LanmanServer key on a typical Windows 2000 Server appear similar to Figure 11-18.
Figure 11-18 LanmanServer registry parameters - before using NET CONFIG command
Chapter 11. Microsoft Windows Server
365
After issuing this single, relatively simple command, however, Figure 11-19 shows the number of registry entries that have been inserted into the LanmanServer key. Each of one of these values is now statically set at the values displayed and is no longer able to be automatically tuned by Windows. Some of these values control important performance and scalability parameters of the Server service. They are now constrained to these static values and cannot be changed unless they are modified manually or remove altogether.
Figure 11-19 LanmanServer registry parameters - after using NET CONFIG command
As shown in these figures, the undesirable impact of typing these commands. To avoid this issue, you need to create the values of autodisconnect, disc, srvcomment, and hidden manually as required when using the registry directly.
366
Tuning IBM System x Servers for Performance
Windows Server 2003 does not populate the LanmanServer key with as many values as in Windows 2000 Server or Windows NT Server, but still introduces several parameters that are best left to auto-tune under most circumstances. This issue is clearly outlined in Microsoft KB 128167. http://support.microsoft.com/?kbid=128167
11.18.7 Monitor system performance appropriately Running the Windows System (Performance) Monitor directly on the server console of a server you are monitoring will impact the performance of the server and will potentially distort the results you are examining. Wherever possible, use System Monitor from a remote system to avoid placing this extra load on the server itself. Similarly, do not use remote control software of the server console for performance monitoring nor should you use a remote client session using thin-client sessions such as Terminal Services or Citrix to carry out the task. Note however that monitoring a system remotely will affect the network and network interface counters which might impact your measurements. Bear in mind that the more counters monitored, the more overhead is required. Don’t monitor unnecessary counters. This tact should be taken regardless of whether carrying out “live” charted monitoring of a system or logging system performance over longer periods. In the same way, do not publish System Monitor as a Citrix application. Along the same lines, reduce the monitoring interval to the minimum necessary to obtain the information you are looking to gather. When logging over an extended period, collecting data every five minutes or even more might still provide acceptable levels of information than the system default of every second. Monitoring more frequently will generate additional system overhead and create significantly larger log files than might be required.
11.19 The Future of Windows Server This section discusses the current plans for follow-on products to Windows Server 2003.
Chapter 11. Microsoft Windows Server
367
11.19.1 Windows Server 2003, Service Pack 2 Windows continually seeks to improve its operating system through the issuing of Service Packs, which are a cumulative roll-up of all previous updates and fixes, including any previous service packs. While the beta refresh 2 has just been realized at time of writing, the release date for the full version of Windows Server 2003, Service Pack 2 is currently scheduled for the first half of 2007. Service Pack 2 will be applicable for both the Windows Server 2003 32-bit (x86) and 64-bit (x64) editions. You can find the release notes for Service Pack 2 as it is released at: http://support.microsoft.com/kb/914961 You can find the roadmap for Windows operating system Service Packs at: http://www.microsoft.com/windows/lifecycle/servicepacks.mspx
11.19.2 Windows Server “Longhorn” Microsoft Windows Server code name “Longhorn” is the next-generation of the Windows Server operating system. It is presently scheduled for release in late 2007, though its widespread adoption is not expected until well into 2008. Longhorn is very significant upgrade of the Windows Server operating system. With many upgrades and changes, it is out of the scope of this chapter to discuss all the differences between Windows Server 2003 and Longhorn, especially in light of the fact that the product is still under final development and thus still subject to change. What is discussed briefly here however are those changes that should offer real server performance improvements. The increased reliability, scalability, security and performance of Windows Server “Longhorn” make it a very impressive upgrade from its predecessors, including Windows Server 2003. Longhorn has been particularly tuned to deliver the best possible performance and integration with the last Windows desktop operating systems, most notably Vista, due for release in late 2006. It is noteworthy that Longhorn is planned to be the last 32-bit release of Windows Server. Windows Server Longhorn R2 is planned to be a 64-bit operating system only.
368
Tuning IBM System x Servers for Performance
Some notable performance changes in Longhorn include: Server roles. As it is shipped, Longhorn is very cut-back in functionality. You need to enable all server roles except file-serving as additional functions for a given server implementation. This functionality serves to tighten security and to improve performance because fewer idle processes are running on the server. Server core. Administrators can choose to install Windows Server with only core server functionality and without any extra overhead. This server core functionality limits the roles a server can perform but reduces management and improves system responsiveness. Server core is a minimal server installation option that provides an environment for running the following server roles: – – – –
Dynamic Host Configuration Protocol (DHCP) Server Domain Name Systems (DNS) Server File Server Domain Controller
The server core installation only installs a subset of the components of a “full” server operating system install. This includes removing the Windows Explorer GUI (shell), leaving only the command prompt to interface with the server. IPv6. Native IPv6 support across all client and server services creates a more scalable and reliable network, while the rewritten TCP/IP stack makes network communication much faster and more efficient. SMB 2.0. The new Server Message Block 2.0 protocol provides a number of communication enhancements, including greater performance when connecting to file shares over high-latency links and better security through the use of mutual authentication and message signing. Memory Manager enhancements. Longhorn includes enhanced support for NUMA (Non-Uniform Memory Architecture), allowing access to “closer” memory nodes, improving performance. Much faster large memory page allocations in both kernel AND user mode will be available. Indexed Search. Searching Windows Server “Longhorn” servers from a Windows Vista client avails of enhanced indexing and caching technologies on both to provide huge performance gains across the enterprise. DNS background zone loading. With Windows Server 2003, Active Directory integrated DNS servers for organizations with very large DNS zones can often take a long time to start as DNS data is retrieved from Active Directory, leaving the server unable to respond to user requests. In Longhorn, DNS loads zone data in the background so it can begin responding to user requests more quickly. It can query AD directly for any node record that might be unavailable while the zone loads directly onto the server.
Chapter 11. Microsoft Windows Server
369
Improved backup technology. The use of Volume Shadow Copy Service and block-level backup technology improves performance and efficiency. Windows Reliability & Performance Monitor. This is a new MMC that combines the functionality of previous tools such as Performance Logs & Alerts, Server Performance Advisor and System Monitor. Windows Systems Resource Manager (WSRM). WSRM allows the server administrator to control how CPU and memory resources are allocated to applications, services and processes running on the server. Managing resources in this manner will offer improvements to system performance and reduces the opportunity for one application, service or process to take system resources away from another. Internet Information Services 7.0 (IIS 7.0). Improvements to the management tools and core feature set of IIS 7.0 yield definite performance benefits to those using IIS as a Web server. Read-only domain controller (RODC). Primarily aimed as a means of improving security for sites where physical security of domain controllers, cannot be guaranteed, the RODC also offers notable performance advantages to the server and wide-area-network (WAN). Local users experience faster logon time and more efficient access to WAN resources.
11.19.3 The 64-logical thread limitation As the number of cores per physical processor increases, on large servers, the limit of 64 logical threads in Windows Server 2003 is becoming restrictive. Large systems, especially those with multiple nodes with multiple dual or even quad core processes and Hyper-Threading enabled have reached a threshold that is imposed by the operating system rather than the hardware. An example of this is the IBM System x3950 systems which can host up to eight nodes each with four dual core processors with Hyper-Threading enabled. This configuration results in 128 logical processors which Windows Server 2003 cannot presently work around. While there is no clear direction on a solution yet from Microsoft, they are well aware of this restriction and are prototyping support for more than 64 logical threads. This functionality will almost certainly not be delivered in the first release of Longhorn. Solving this particular problem is a particularly difficult one and means tackling many compatibility issues for applications and device drivers. For more information, see the following PowerPoint® presentation: http://download.microsoft.com/download/5/b/9/5b97017b-e28a-4bae-ba48-17 4cf47d23cd/SER058_WH06.ppt
370
Tuning IBM System x Servers for Performance
12
Chapter 12.
Linux By its nature and heritage, the Linux distributions and the Linux kernel offer a variety of parameters and settings to let the Linux administrator tweak the system to maximize performance. This chapter describes the steps that you can take to tune Red Hat Enterprise Linux (RHEL)1 or SUSE Linux Enterprise Server (SLES)2. It describes the parameters that give the most improvement in performance and provides a basic understanding of the tuning techniques that are used in Linux. In this chapter, we discuss the following topics: 1 2
12.1, “Disabling daemons” on page 372 12.2, “Shutting down the GUI” on page 376 12.3, “SELinux (RHEL 4 only)” on page 379 12.4, “Changing kernel parameters” on page 381 12.5, “Kernel parameters” on page 384 12.6, “Tuning the processor subsystem” on page 388 12.7, “Tuning the memory subsystem” on page 391 12.8, “Tuning the file system” on page 395 12.9, “Tuning the network subsystem” on page 407 12.10, “SUSE Linux Enterprise Server 10” on page 411 12.11, “Xen virtualization” on page 420 Red Hat is a registered trademark of Red Hat, Inc. The information and screen captures in this chapter were used with permission. SUSE Linux is a registered trademark of SUSE Linux AG, a Novell, Inc. company. The information and screen captures in this chapter were used with permission from Novell, Inc.
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
371
12.1 Disabling daemons There are daemons (background services) running on every server that are probably not needed. Disabling these daemons frees memory, decreases startup time, and decreases the number of processes that the CPU has to handle. A side benefit is increased security of the server because fewer daemons mean fewer exploitable processes. By default, several daemons are started that can be stopped safely on most servers. Table 12-1 lists the daemons that apply to Red Hat Enterprise Linux AS 3 and 4. Table 12-1 Red Hat Enterprise Linux AS: Tunable daemons started by default
Daemons
RHEL 3
RHEL 4
Description
apmd
Yes
Yes
Advanced power management daemon. apmd, which is most likely used on a server.
arptables_jf
No
Yes
User space program for the arptables network filter. Unless you plan to use arptables, you can safely disable this daemon.
autofs
Yes
Yes
Automatically mounts file systems on demand (for example, mounts a CD-ROM automatically). On server systems, file systems rarely have to be mounted automatically.
cpuspeed
No
Yes
Daemon to adjust the frequency of the CPU dynamically. In a server environment, this daemon is recommended off.
cups
Yes
Yes
Common UNIX Printing System. If you plan to provide print services with your server, do not disable this service.
gpm
No
No
Mouse server for the text console. Do not disable if you want mouse support for the local text console.
hpoj
Yes
No
HP OfficeJet support. Do not disable if you plan to use an HP OfficeJet printer with your server.
irqbalance
No
Yes
Balances interrupts between multiple processors. You can safely disable this daemon on a singe CPU system or if you plan to balance IRQ statically.
isdn
Yes
Yes
ISDN modem support. Do not disable if you plan to use an ISDN modern with your server.
kudzu
Yes
Yes
Detects and configures new hardware. Should be run manually in case of a hardware change.
netfs
Yes
Yes
Used in support of exporting NFS shares. Do not disable if you plan to provide NFS shares with your server.
372
Tuning IBM System x Servers for Performance
Daemons
RHEL 3
RHEL 4
Description
nfslock
Yes
Yes
Used for file locking with NFS. Do not disable if you plan to provide NFS shares with your server.
pcmcia
Yes
Yes
PCMCIA support on a server. Server systems rarely rely on a PCMCIA adapter, so disabling this daemon is safe in most instances.
portmap
Yes
Yes
Dynamic port assignment for RPC services (such as NIS and NFS). If the system does not provide RPC-based services there is no need for this daemon.
rawdevices
No
Yes
Provides support for raw device bindings. If you do not intend to use raw devices, you can safely turn it off.
rpc
No
Yes
Various remote procedure call daemons used mainly for NFS and Samba. If the system does not provide RPC-based services, there is no need for this daemon.
sendmail
Yes
Yes
Mail Transport Agent. Do not disable this daemon if you plan to provide mail services with the respective system.
smartd
No
Yes
Self Monitor and Reporting Technology daemon that watches S.M.A.R.T. compatible devices for errors. Unless you use an IDE/ SATA technology–based disk subsystem, there is no need for S.M.A.R.T. Monitoring.
xfs
Yes
Yes
Font server for X-Window Systems. If you will run in run level 5, do not disable this daemon.
Table 12-2 lists the daemons that apply to SUSE Linux Enterprise Server. Table 12-2 SUSE Linux Enterprise Server: Tunable daemons started by default
Daemon
Description
alsasound
Sound daemon
isdn
ISDN modem support
hwscan
Detects and configures hardware changes
portmap
Dynamic port assignment for RPC services (such as NIS and NFS)
postfix
Mail Transport Agent
splash
Splash screen setup
fbset
Framebuffer setup
splash_late
Starts before shutdown
splash_early
Kills animation after the network starts
xdm
X Display manager
Chapter 12. Linux
373
Important: Turing off the xfs daemon prevents X from starting on the server. So, you should only turn off this if the server will not be booting into the GUI. Simply starting the xfs daemon before issuing the startx command enables X to start normally. You can stop a daemon immediately if required. For example, you can stop the sendmail daemon using the following commands: RHEL: /sbin/service sendmail stop SLES: /etc/init.d/sendmail stop You can use the status parameter to see whether a daemon is running, and the start parameter to start the daemon. You can also configure the daemon to not start the next time the server starts. Again, for the sendmail daemon, you can use the following commands: RHEL: /sbin/chkconfig sendmail off SLES: /sbin/chkconfig -s sendmail off In addition, these distributions provide a graphical interface to managing daemons. Tip: People often think that changes performed through chkconfig are not active until the next reboot. Actually changing the run level has the same effect on the running daemons as rebooting does. Instead of wasting precious time waiting for a reboot to complete, simply change the run level to 1 and back to 3 or 5, respectively). To run the GUI, issue the following command: RHEL 3: /usr/bin/redhat-config-services RHEL 4: /usr/bin/system-config-services
374
Tuning IBM System x Servers for Performance
Alternatively, you can click Main Menu → System Settings → Server Settings → Services. The Red Hat Service Configuration window opens, as shown in Figure 12-1.
To change the current state, highlight the daemon, and click Stop. The check mark indicates that the daemon will start at the next reboot.
Figure 12-1 Red Hat Service Configuration interface
Tip: Not all daemons appear in the Red Hat Service Configuration window. To see a complete list, issue the following command: /sbin/chkconfig --list For SLES, the graphical interface is YaST2, which you can start either with the command /sbin/yast2 runlevel or by clicking Browse: YaST/ → YaST modules → System → Runlevel editor, as shown in Example 12-2 on page 376.
Chapter 12. Linux
375
Figure 12-2 SUSE Linux YaST runlevel editor
12.2 Shutting down the GUI Whenever possible, do not run the GUI on a Linux server. Normally, there is no need for a GUI on a Linux server. You can perform all administration tasks through the command line by redirecting the X display or through a Web browser interface. There are several different useful Web-based tools (for example, webmin, Linuxconf, and SWAT). If you must use a GUI, start and stop it as needed rather than running it all the time. In most cases the server should be running at run level 3, which does not start the X Server when the machine boots up. If you want to restart the X Server, use the startx command from a command prompt. Follow these steps: 1. Determine at which run level the machine is running by using the runlevel command. This command prints the previous and current run level. For example, N 5 means that there was no previous run level (N) and that the current run level is 5.
376
Tuning IBM System x Servers for Performance
2. To switch between run levels, use the init command. (For example, to switch to run level 3, enter init 3. The run levels that are used in Linux are: 0
Halt (Do not set initdefault to this or the server will shut down immediately after finishing the boot process.)
1
Single user mode
2
Multiuser, without NFS (the same as 3, if you do not have networking)
3
Full multiuser mode
4
Unused
5
X11
6
Reboot (Do not set initdefault to this or the server machine will continuously reboot at startup.)
Chapter 12. Linux
377
To set the initial run level of a machine at boot, modify the /etc/inittab file as shown in Figure 12-3 with the following line: id:3:initdefault: ... (lines not displayed)
To start Linux without starting the GUI, set the run level to 3.
# The default runlevel is defined here id:3:initdefault: # First script to be executed, if not booting in emergency (-b) mode si::bootwait:/etc/init.d/boot # # # # # # # # # #
/etc/init.d/rc takes care of runlevel handling runlevel runlevel runlevel runlevel runlevel runlevel runlevel
0 1 2 3 4 5 6
is is is is is is is
System halt (Do not use this for initdefault!) Single user mode Local multiuser without remote network (e.g. NFS) Full multiuser with network Not used Full multiuser with network and xdm System reboot (Do not use this for initdefault!)
... (lines not displayed) # getty-programs for the normal runlevels # :::<process> # The “id” field MUST be the same as the last # characters of the device (after “tty”). 1:2345:respawn:/sbin/mingetty --noclear tty1 2:2345:respawn:/sbin/mingetty tty2 3:2345:respawn:/sbin/mingetty tty3 #4:2345:respawn:/sbin/mingetty tty4 #5:2345:respawn:/sbin/mingetty tty5 #6:2345:respawn:/sbin/mingetty tty6 # #S0:12345:respawn:/sbin/agetty -L 9600 ttyS0 vt102 ... (lines not displayed) Figure 12-3 /etc/inittab, modified (only part of the file is displayed)
378
Tuning IBM System x Servers for Performance
To only provide three consoles and thereby save memory, comment out the mingetty entries for 4, 5, and 6.
With SUSE Linux Enterprise Server, shutting down the GUI can also be accomplished by running the YaST runlevel command and changing the default run level (see Figure 12-2 on page 376). By default, six consoles are saved. F1 through F6 are separate consoles. To regain some memory, you might want to limit the number of consoles to three from the original six. To do this, comment out each mingetty ttyx line that you want to disable. In Figure 12-3 on page 378, for example, we have limited the consoles to three. Tip: Even if you have the GUI disabled locally on the server, you can still connect remotely and use the GUI, using the -X parameter on the ssh command.
12.3 SELinux (RHEL 4 only) Red Hat Enterprise Linux 4 introduced a new security model, Security Enhanced Linux (SELinux), which is a significant step toward higher security. SELinux introduces a mandatory policy model that overcomes the limitations of the standard discretionary access model employed by Linux. SELinux enforces security on user and process levels. Thus, a security flaw of any given process affects only the resources that are allocated to this process and not the entire system. SELinux works similar to a virtual machine (see Figure 12-4).
Security policy
Process Request access User
Grant access
System resouces
Security enforcement module
SELinux kernel
Grant/Deny access based on policy
Figure 12-4 Schematic overview of SELinux
Chapter 12. Linux
379
For example, if a malicious attacker uses a root exploit within Apache, only the resources that are allocated to the Apache daemon could be compromised. Novell introduced a similar feature in version 10 of its SUSE Linux Enterprise Server (see 12.10, “SUSE Linux Enterprise Server 10” on page 411). Enforcing such a policy-based security model, however, comes at a price. Every access from a user or process to a system resource, such as an I/O device, must be controlled by SELinux. Thus, the process of checking permissions can cause an overhead of up to 10%. SELinux is of great value to any edge server such as a firewall or a Web server, but the added level of security on a back-end database server might not justify the loss in performance. Generally, the easiest way to disable SELinux is to not install it in the first place. Often systems have been installed using default parameters and are unaware that SELinux affects performance. To disable SELinux after an installation, append the selinux=0 entry to the line that includes the running kernel in the GRUB boot loader, as shown in Example 12-1. Example 12-1 Sample grub.conf file with disabled SELinux
default=0 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title Red Hat Enterprise Linux AS (2.6.9-5.ELsmp) root (hd0,0) kernel /vmlinuz-2.6.9-5.ELsmp ro root=LABEL=/ selinux=0 initrd /initrd-2.6.9-5.ELsmp.img If you decide to use SELinux with your Linux-based server, you can tweak its settings to better accommodate your environment. On a running system, check whether the working set of the cached Linux Security Modules (LSM) permissions exceeds the default Access Vector Cache (AVC) size of 512 entries. Check /selinux/avc/hash_stats for the length of the longest chain. Anything over 10 signals a likely bottleneck. Tip: To check for usage statistics of the access vector cache, you can alternatively use the avcstat utility. If the system experiences a bottleneck in the Access Vector Cache (for example, on a heavily loaded firewall), try to resize /selinux/avc/cache_threshold to a slightly higher value and recheck the hash statistics.
380
Tuning IBM System x Servers for Performance
12.4 Changing kernel parameters The Linux kernel is the core of the operating system and is common to all Linux distributions. You can make changes to the kernel by modifying parameters that control the operating system. You make these changes on the command line using the sysctl command. Tip: By default, the kernel includes the necessary module to enable you to make changes using sysctl without needing to reboot. However, If you choose to remove this support (during the operating system installation), then you have to reboot Linux before the change takes effect. SUSE Linux offers a graphical method of modifying these sysctl parameters. To launch the powertweak tool, issue the following command: /sbin/yast2 powertweak For a text-based menu version, use the command: /sbin/yast powertweak
Figure 12-5 SUSE Linux powertweak
Chapter 12. Linux
381
12.4.1 Parameter storage locations The kernel parameters that control how the kernel behaves are stored in /proc (and in particular, /proc/sys). Reading the files in the /proc directory tree provides a simple way to view configuration parameters that are related to the kernel, processes, memory, network, and other components. Each process running in the system has a directory in /proc with the process ID (PID) as name. Table 12-3 lists some of the files that include kernel information. Table 12-3 Parameter files in /proc
Files/directory
Purpose
/proc/loadavg
Information about the load of the server in 1-minute, 5-minute, and 15-minute intervals. The uptime command gets information from this file.
/proc/kcore
(SUSE Linux Enterprise Server only) includes data to generate a core dump at run time, for kernel debugging purposes. The command to create the core dump is gdb as in the #gdb /usr/src/linux/vmlinux /proc/kcore directory.
/proc/stat
Kernel statistics as process, swap and disk I/O.
/proc/cpuinfo
Information about the installed CPUs.
/proc/meminfo
Information about memory usage. The free command uses this information.
/proc/sys/abi/* (all files in this directory)
Used to provide support for binaries that are not native to Linux and that are compiled under other UNIX variants, such as SCO UnixWare 7, SCO OpenServer, and SUN Solaris™ 2. By default, this support is installed, although it can be removed during installation.
/proc/sys/fs/*
Used to increase the number of open files that the operating system allows and to handle quota.
/proc/sys/kernel/*
For tuning purposes, you can enable hotplug, manipulate shared memory, and specify the maximum number of PID files and level of debug in syslog.
/proc/sys/net/*
Tuning of network in general, IPV4 and IPV6.
/proc/sys/vm/*
Management of cache memory and buffer.
382
Tuning IBM System x Servers for Performance
12.4.2 Using the sysctl commands The sysctl commands use the names of files in the /proc/sys directory tree as parameters. For example, to modify the shmmax kernel parameter, you can display (using cat) and change (using echo) the file /proc/sys/kernel/shmmax as in the following example: #cat /proc/sys/kernel/shmmax 33554432 #echo 33554430 > /proc/sys/kernel/shmmax #cat /proc/sys/kernel/shmmax 33554430 However, using these commands can introduce errors easily. So, we recommend that you use the sysctl command because it checks the consistency of the data before it makes any change. For example: #sysctl kernel.shmmax kernel.shmmax = 33554432 #sysctl -w kernel.shmmax=33554430 kernel.shmmax = 33554430 #sysctl kernel.shmmax kernel.shmmax = 33554430 This change to the kernel stays in effect only until the next reboot. If you want to make the change permanent, edit the /etc/sysctl.conf or /etc/sysconfig/sysctl file and add the appropriate command. In our example: kernel.shmmax = 33554439 The next time you reboot, the parameter file is read. You can do the same thing without rebooting by issuing the following command: #sysctl -p
Chapter 12. Linux
383
12.5 Kernel parameters The Linux kernel has many parameters that can improve performance for your installation. Table 12-4 lists the SLES kernel parameters that are most relevant to performance. Table 12-4 SLES parameters that are most relevant to performance tuning
Parameter
Description / example of use
kernel.shm-bigpages-per-file
Normally used for tuning database servers. The default is 32768. To calculate a suitable value, take the amount of SGA memory in GB and multiply by 1024. For example: sysctl -w kernel.shm-bigpages-per-file=16384
kernel.sched_yield_scale
Enables the dynamic resizing of time slices given to processes. When enabled, the kernel reserves more time slices for busy processes and fewer for idle processes. The parameters kernel.min-timeslice and kernel.max-timeslice are used to specify the range of time slices that the kernel can supply as needed. If disabled, the time slices given to each process are the same. The default is 0 (disabled). Applications such as ERP and Java can benefit from this being enabled. For real-time applications such as streaming audio and video, leave it disabled. For example: sysctl -w kernel.sched_yield_scale=1
kernel.shm-use-bigpages
Enables the use of bigpages (typically for databases). Default is 0 (disabled). For example: sysctl -w kernel.shm-use-bigpages=1
net.ipv4.conf.all.hidden
All interface addresses are hidden from ARP broadcasts and will be included in the ARP response of other addresses. Default is 0 (disabled). For example: sysctl -w net.ipv4.conf.all.hidden=1
net.ipv4.conf.default.hidden
Enables all interfaces as hidden by default. Default is 0 (disabled). sysctl -w net.ipv4.conf.default.hidden=1
net.ipv4.conf.eth0.hidden
Enables only interface eth0 as hidden. Uses the ID of your network card. Default is 0 (disabled). sysctl -w net.ipv4.conf.eth0.hidden=1
net.ipv4.ip_conntrack_max
This setting is the number of separate connections that can be tracked. Default is 65536. sysctl -w net.ipv4.ip_conntrack_max=32768
net.ipv6.conf.all.mtu
Default maximum for transfer unit on IPV6. Default is 1280. sysctl -w net.ipv6.conf.all.mtu=9000
384
Tuning IBM System x Servers for Performance
Parameter
Description / example of use
net.ipv6.conf.all. router_solicitation_delay
Determines whether to wait after interface opens before sending router solicitations. Default is 1 (the kernel should wait). For example: sysctl -w net.ipv6.conf.all.router_solicitation_delay=0
net.ipv6.conf.all. router_solicitation_interval
Number of seconds to wait between router solicitations. Default is 4 seconds. For example: sysctl -w net.ipv6.conf.all.router_solicitation_interval=3
net.ipv6.conf.all. router_solicitations
Number of router solicitations to send until assuming no routers are present. Default is 3. sysctl -w net.ipv6.conf.all.router_solicitations=2
net.ipv6.conf.all. temp_prefered_lft
Lifetime preferred in seconds for temporary addresses. Default is 86400 (1 day). sysctl -w net.ipv6.conf.all.temp_prefered_lft=259200
net.ipv6.conf.all. temp_valid_lft
Lifetime valid in seconds for temporary address. Default is 604800 (1 week). sysctl -w net.ipv6.conf.all.temp_valid_lft=302400
net.ipv6.conf.default. accept_redirects
Accepts redirects sent by a router that works with IPV6, but it cannot set if forwarding is set to enable. Always one or other, it can never set together because it will cause problems in all-IPV6 networks. Default is 1 (enabled). sysctl -w net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.default. autoconf
This automatically generates an address such as "ff81::221:21ff:ae44:2781" on an interface with an L2-MAC Address. Default is 1 (enabled). sysctl -w net.ipv6.conf.default.autoconf=0
net.ipv6.conf.default. dad_transmits
Determines whether Duplicate Address Detection (DAD) probes are sent. Default is 1 (enabled). sysctl -w net.ipv6.conf.default.dad_transmits=0
net.ipv6.conf.default.mtu
Sets the default value for Maximum Transmission Unit (MTU). Default is 1280. sysctl -w net.ipv6.conf.default.mtu=9000
net.ipv6.conf.default. regen_max_retry
Number of attempts to try to generate a valid temporary address. Default is 5. sysctl -w net.ipv6.conf.default.regen_max_retry=3
net.ipv6.conf.default. router_solicitation_delay
Number in seconds to wait, after interface is brought up, before sending router request. Default is 1 (enabled). sysctl -w net.ipv6.conf.default.router_solicitation_delay=0
Chapter 12. Linux
385
Parameter
Description / example of use
vm.heap-stack-gap
Enables the heap of memory that is used to store information about status of process and local variables. You should disable this when you need to run a server with JDK™; otherwise, your software will crash. Default is 1 (enabled). sysctl -w vm.heap-stack-gap=0
vm.vm_anon_lru
Allows the VM to always have visibility of anonymous pages. Default is 1 (enabled). sysctl -w vm.vm_anon_lru=0
vm.vm_lru_balance_ratio
Balances active and inactive sections of memory. Define the amount of inactive memory that the kernel will rotate. Default is 2. sysctl -w vm.vm_lru_balance_ratio=3
vm.vm_mapped_ratio
Controls the pageout rate. Default is 100. sysctl -w vm.vm_mapped_ratio=90
vm.vm_passes
Number of attempts that the kernel will try to balance the active and inactive sections of memory. Default is 60. sysctl -w vm.vm_passes=30
vm.vm_shmem_swap
Improves performance of applications that use large amounts of non-locked shared memory (such as ERP and database applications) on a server with more than 8 GB of RAM. Default is 0 (disabled). sysctl -w vm.vm_shmem_swap=1
vm.vm_vfs_scan_ratio
Proportion of Virtual File System unused caches that will try to be in one VM freeing pass. Default is 6. sysctl -w vm.vm_vfs_scan_ratio=6
386
Tuning IBM System x Servers for Performance
Table 12-5 lists the RHEL kernel parameters that are most relevant to performance. Table 12-5 Red Hat parameters that are most relevant to performance tuning
Parameter
Description / example of use
net.ipv4. inet_peer_gc_maxtime
How often the garbage collector (gc) should pass over the inet peer storage memory pool during low or absent memory pressure. Default is 120, measured in jiffies. sysctl -w net.ipv4.inet_peer_gc_maxtime=240
net.ipv4. inet_peer_gc_mintime
Sets the minimum time that the garbage collector can pass cleaning memory. If your server is heavily loaded, you might want to increase this value. Default is 10, measured in jiffies. sysctl -w net.ipv4.inet_peer_gc_mintime=80
net.ipv4.inet_peer_maxttl
The maximum time-to-live for the inet peer entries. New entries will expire after this period of time. Default is 600, measured in jiffies. sysctl -w net.ipv4.inet_peer_maxttl=500
net.ipv4.inet_peer_minttl
The minimum time-to-live for inet peer entries. Set to a high enough value to cover fragment time to live in the reassembling side of fragmented packets. This minimum time must be smaller than net.ipv4.inet_peer_threshold. Default is 120, measured in jiffies. sysctl -w net.ipv4.inet_peer_minttl=80
net.ipv4. inet_peer_threshold
Set the size of inet peer storage. When this limit is reached, peer entries will be thrown away, using the inet_peer_gc_mintime timeout. Default is 65644. sysctl -w net.ipv4.inet_peer_threshold=65644
vm.hugetlb_pool
The hugetlb feature works in the same way as bigpages, but after hugetlb allocates memory, only the physical memory can be accessed by hugetlb or shm allocated with SHM_HUGETLB. It is normally used with databases such as Oracle or DB2. Default is 0. sysctl -w vm.hugetlb_pool=4608
vm.inactive_clean_percent
Designates the percent of inactive memory that should be cleaned. Default is 5%. sysctl -w vm.inactive_clean_percent=30
vm.pagecache
Designates how much memory should be used for page cache. This is important for databases such as Oracle and DB2. Default is 1 15 100. This parameter’s three values are: Minimum percent of memory used for page cache. Default is 1%. The initial amount of memory for cache. Default is 15%. Maximum percent of memory used for page cache. Default is 100%. sysctl -w vm.pagecache=1 50 100
Chapter 12. Linux
387
12.6 Tuning the processor subsystem The CPU is one of the most important hardware subsystems for servers whose primary role is that of an application or database server. However, in these systems, the CPU is often the source of performance bottlenecks. For information about the tweaking of processor tuning parameters, refer to 12.5, “Kernel parameters” on page 384. On high-end servers with Xeon processors, you might want to enable or disable Hyper-Threading. Hyper-Threading is a way of virtualizing each physical processor as two processors to the operating system, and is supported by both Red Hat Enterprise Linux AS and SUSE Linux Enterprise Server. By virtualizing the processor, you can execute two threads or processes at a time (this is also known as thread-level parallelism). By having your operating system and software designed to take advantage of this technology, you can gain significant increases in performance without needing an increase in clock speed. For example, if you have Hyper-Threading enabled on a 4-way server, monitoring tools such as top will display eight processors (Example 12-2). Example 12-2 Output of top on a 4-way server with Hyper-Threading enabled
10:22:45 up 23:40, 5 users, load average: 26.49, 12.03, 10.24 373 processes: 370 sleeping, 2 running, 1 zombie, 0 stopped CPU states: cpu user nice system irq softirq iowait idle total 36.1% 0.1% 9.7% 0.3% 4.1% 1.6% 47.7% cpu00 17.0% 0.0% 5.9% 3.1% 20.8% 2.1% 50.7% cpu01 54.9% 0.0% 10.9% 0.0% 0.9% 1.3% 31.7% cpu02 33.4% 0.1% 8.5% 0.0% 2.5% 0.9% 54.2% cpu03 33.8% 0.7% 10.0% 0.0% 0.9% 2.1% 52.0% cpu04 31.4% 0.0% 9.3% 0.0% 2.9% 2.5% 53.6% cpu05 33.4% 0.0% 9.9% 0.0% 2.1% 0.7% 53.6% cpu06 30.5% 0.0% 11.1% 0.0% 1.7% 1.3% 55.1% cpu07 54.5% 0.0% 12.1% 0.0% 0.5% 1.9% 30.7% Mem: 8244772k av, 3197880k used, 5046892k free, 0k shrd, 91940k buff 2458344k active, 34604k inactive Swap: 2040244k av, 0k used, 2040244k free 1868016k cached
388
Tuning IBM System x Servers for Performance
With respect to Hyper-Threading, note that: SMP-based kernels are required to support Hyper-Threading. The more CPUs that are installed in a server, the fewer benefits Hyper-Threading has on performance. On servers that are CPU-bound, expect, at most, the following performance gains: – Two physical processors: 15% to 25% performance gain – Four physical processors: 15 to 13% gain – Eight physical processors: 0 to 5% gain Note: These performance gains are true for specific workloads, software, and operating system combinations only. For more information about Hyper-Threading, see: http://www.intel.com/business/bss/products/hyperthreading/server/ EM64T is a 64-bit extension to Intel IA-32 processors, which means that the processors are capable of addressing more memory and can support new 64-bit applications while remaining fully compatible with all existing 32-bit applications. Support for this new processor is in Red Hat Enterprise Linux 3 Update 2 and SUSE Linux Enterprise Server 9. For more information about EM64T, see: http://www.intel.com/technology/64bitextensions/
12.6.1 Selecting the correct kernel Both Red Hat Enterprise Linux AS and SUSE Linux Enterprise Server include several kernel packages, as listed in Table 12-6. It is important for performance reasons that you select the most appropriate kernel for your system. Table 12-6 Available kernels within the distribution
Kernel type
Description
SMP
Kernel has support for SMP and Hyper-Threaded machines.
Hugemem
(Red Hat Enterprise Linux AS only) Support for machines with greater than 12 GB of memory. Includes support for NUMA.
Standard
Single processor machines.
Chapter 12. Linux
389
12.6.2 Interrupt handling One of the highest-priority tasks a CPU has to handle is interrupts. Interrupts can be caused by subsystems, such as a network interface card. Hard interrupts cause a CPU to stop its current work and perform a context switch, which is undesirable because the processor has to flush its cache to make room for the new work. (Think of a processor’s cache as a work bench that has to be cleaned up and supplied with new tools every time new work has to be done.) Two principles have proven to be most efficient when it comes to interrupt handling: Bind processes that cause a significant amount of interrupts to a specific CPU. CPU affinity enables the system administrator to bind interrupts to a group or a single physical processor (of course, this does not apply on a single-CPU system). To change the affinity of any given IRQ, go into /proc/irq/%{number of respective irq}/ and change the CPU mask stored in the file smp_affinity. For example, to set the affinity of IRQ 19 to the third CPU in a system (without Hyper-Threading) use the following command: echo 03 > /proc/irq/19/smp_affinity Let physical processors handle interrupts. In Hyper-Threading-enabled systems such as Intel Xeon processors, it is suggested that you bind interrupt handling to the physical processor rather than the Hyper-Threading instance. The physical processors usually have the lower CPU numbering. So, in a 2-way system with Hyper-Threading enabled, CPU ID 0 and 2 would refer to the physical CPU, and 1 and 3 would refer to the Hyper-Threading instances. Note: For multi-core sockets, each core is seen by the operating system as one processor. Each core has its own cache L1, meaning that it does not share its cache (at least, this layer) with another processor. Then, the physical processor already handles interrupts. Hyper-Threading is not supported on multi-core sockets.
Considerations for NUMA systems Non-uniform memory access (NUMA) systems are gaining market share and are seen as the natural evolution of classic symmetric multiprocessor systems. Although the CPU scheduler used by current Linux distributions is well suited for NUMA systems, applications might not always be. Bottlenecks caused by a non-NUMA-aware application can cause performance degradations that are hard to identify. The recent numastat utility shipped in the numactl package helps to identify processes that have difficulties dealing with NUMA architectures.
390
Tuning IBM System x Servers for Performance
To help with spotting bottlenecks, you can use the statistics that are provided by the numastat tool in the /sys/devices/system/node/%{node number}/numastat file. High values in numa_miss and the other_node field signal a likely NUMA issue. If you find that a process is allocated memory that does not reside on the local node for the process (the node that holds the processors that run the application), try to renice the process to the other node or work with NUMA affinity.
12.7 Tuning the memory subsystem Tuning the memory subsystem is a difficult task that requires constant monitoring to ensure that changes do not negatively affect other subsystems in the server. If you do choose to modify the virtual memory parameters (in /proc/sys/vm), we recommend that you change only one parameter at a time and monitor how the server performs. Remember that most applications under Linux do not write directly to the disk, but to the file system cache that is maintained by the virtual memory manager that will eventually flush out the data. When using an IBM ServeRAID controller or an IBM TotalStorage disk subsystem, you should try to decrease the number of flushes, effectively increasing the I/O stream that is caused by each flush. The high-performance disk controller can handle the larger I/O stream more efficiently than multiple small ones.
12.7.1 Configuring bdflush (kernel 2.4 only) There is tuning in the virtual memory subsystem that can help improve overall file system performance. The bdflush kernel daemon is responsible for making sure that dirty buffers, any modified data that currently resides only in the volatile system memory, are committed to disk. Changes in the /proc system take effect immediately but will be reset at boot time. To make changes permanent, include the echo or sysctl command in the /etc/rc.d/rc.local file. Configuring how the Linux kernel flushes dirty buffers to disk can tailor the flushing algorithm toward the specifications of the respective disk subsystem. Disk buffers are used to cache data that is stored on disks, which are very slow compared with RAM. So, if the server uses this kind of memory, it can create serious problems with performance.
Chapter 12. Linux
391
By modifying the /proc/sys/vm/bdflush parameters, you can modify the writing-to-disk rate, possibly avoiding disk contention problems. To edit the parameters of the bdflush subsystem, you can use either the echo command as shown in Example 12-3 or sysctl as shown in Example 12-4, although we recommend that you to use sysctl. Example 12-3 Modifying the bdflush parameters in the kernel using echo
echo 30 500 0 0 500 30000 60 20 0 > /proc/sys/vm/bdflush Example 12-4 Using sysctl to change parameters of bdflush
sysctl -w vm.bdflush="30 500 0 0 500 3000 60 20 0" The parameters in the /proc/sys/vm/bdflush of 2.4 Linux kernels are:
392
nfract
Maximum percentage of dirty buffers in the buffer cache. The higher the value, the longer the write to the disk will be postponed. When available memory is in short supply, large amounts of I/O have to be processed. To spread I/O out evenly, keep this a low value.
ndirty
Maximum number of dirty buffers that the bdflush process can write to disk at one time. A large value results in I/O occurring in bursts, and a small value might lead to memory shortages if the bdflush daemon is not executed enough.
dummy2
Unused (formerly nrefill).
dummy3
Unused.
interval
Minimum rate at which kupdate will wake and flush. Default is 5 seconds, with a minimum value of zero (0) seconds and a maximum of 600 seconds. kupdate is a daemon that manages in-use memory.
age_buffer
Maximum time the operating system waits before writing buffer cache to disk. Default is 30 seconds, with a minimum of one second and a maximum of 6000 seconds.
nfract_sync
Percent of dirty buffers to activate bdflush synchronously. Default is 60%.
nfract_stop
Percent of dirty buffers to stop bdflush. Default is 20%.
dummy5
Unused.
Tuning IBM System x Servers for Performance
12.7.2 Configuring kswapd (kernel 2.4 only) Another pertinent vm subsystem is the kswapd daemon. You can configure this daemon in order to specify how many pages of memory are paged out by Linux: sysctl -w vm.kswapd="1024 32 64" The three parameters are as follows: tries_base is four times the number of pages that the kernel swaps in one pass. On a system with a lot of swapping, increasing the number might improve performance. tries_min is the minimum number of pages that kswapd swaps out each time the daemon is called. swap_cluster is the number of pages that kswapd writes at the same time. A smaller number increases the number of disk I/Os performed, but a larger number might also have a negative impact on the request queue. If you do make changes, check their impact using tools such as vmstat. Other relevant VM parameters that might improve performance include:
buffermem freepages overcommit_memory page-cluster pagecache pagetable_cache
12.7.3 Setting kernel swap behavior (kernel 2.6 only) With the introduction of the improved virtual memory subsystem in the Linux kernel 2.6, administrators now have a simple interface to fine-tune the swapping behavior of the kernel. You can use the parameters in /proc/sys/vm/swappiness to define how aggressively memory pages are swapped to disk. Linux moves memory pages that have not been accessed for some time to the swap space even if there is enough free memory available. By changing the percentage in /proc/sys/vm/swappiness, you can control that behavior, depending on the system configuration. If swapping is not desired, /proc/sys/vm/swappiness should have low values. Systems with memory constraints that run batch jobs (processes that sleep for a long time) might benefit from an aggressive swapping behavior.
Chapter 12. Linux
393
To change swapping behavior, use either echo or sysctl as shown in Example 12-5. Example 12-5 Changing swappiness behavior
# sysctl -w vm.swappiness=100
12.7.4 HugeTLBfs The HugeTLBfs memory management feature is valuable for applications that use a large virtual address space. It is especially useful for database applications. The CPU’s Translation Lookaside Buffer (TLB) is a small cache used for storing virtual-to- physical mapping information. By using the TLB, a translation can be performed without referencing the in-memory page table entry that maps the virtual address. However, to keep translations as fast as possible, the TLB is typically quite small. It is not uncommon for large memory applications to exceed the mapping capacity of the TLB. The HugeTLBfs feature permits an application to use a much larger page size than normal, so that a single TLB entry can map a correspondingly larger address space. A HugeTLB entry can vary in size. For example, in an Itanium 2 system, a huge page might be 1000 times larger than a normal page. This enables the TLB to map 1000 times the virtual address space of a normal process without incurring a TLB cache miss. For simplicity, this feature is exposed to applications by means of a file system interface. Important: Although there are good tools to tune the memory subsystem, swapping should be avoided as much as possible. The fact that the server swaps is almost never a good behavior. Before trying to improve the swap process, ensure that your server simply has enough memory or that there is no memory leak.
394
Tuning IBM System x Servers for Performance
12.8 Tuning the file system Ultimately, all data must be retrieved from and stored to disk. Disk accesses are usually measured in milliseconds and are thousands of times slower than other components (such as memory or PCI operations, which are measured in nanoseconds or microseconds). The Linux file system is the method by which data is stored and managed on the disks. Many different file systems are available for Linux that differ in performance and scalability. In addition to storing and managing data on the disks, file systems are also responsible for guaranteeing data integrity. The newer Linux distributions include journaling file systems as part of their default installation. Journaling or logging prevents data inconsistency in case of a system crash. All modifications to the file system metadata have been maintained in a separate journal or log and can be applied after a system crash to bring it back to its consistent state. Journaling also improves recovery time, because there is no need to perform file system checks at system reboot. As with other aspects of computing, you will find that there is a trade-off between performance and integrity. However, as Linux servers make their way into corporate data centers and enterprise environments, requirements such as high availability can be addressed. In this section, we discuss the default file systems that are available on Red Hat Enterprise Linux AS and SUSE Linux Enterprise Server and some simple ways to improve their performance.
12.8.1 Hardware considerations before installing Linux Minimum requirements for CPU speed and memory are well documented for current Linux distributions. Those instructions also provide guidance for the minimum disk space that is required to complete the installation. However, they fall short when detailing how to set up the disk subsystem initially. Because Linux servers cover a vast assortment of work environments as server consolidation makes its impact in data centers, one of the first questions to answer is: What is the function of the server that is being installed? A server’s disk subsystems can be a major component of overall system performance. Understanding the function of the server is key to determining whether the I/O subsystem will have a direct impact on performance.
Chapter 12. Linux
395
The following examples show where disk I/O is most important: A file and print server must move data quickly between users and disk subsystems. Because the purpose of a file server is to deliver files to the client, the server must initially read all data from a disk. A database server’s ultimate goal is to search and retrieve data from a repository on the disk. Even with sufficient memory, most database servers will perform large amounts of disk I/O to bring data records into memory and flush modified data to disk. The following examples show where disk I/O is not the most important subsystem: An e-mail server acts as a repository and router for electronic mail and tends to generate a heavy communication load. Networking is more important for this type of server. A Web server that is responsible for hosting Web pages (static, dynamic, or both) benefits from a well-tuned network and memory subsystem.
Disk technology selection In addition to understanding the function of the server, you must also understand the size of the deployment that the installation will have to serve. Current disk subsystem technologies were designed with size of deployment in mind. See Table 9-1 on page 170 for a brief description of the disk technologies.
Number of drives The number of disk drives affects performance significantly because each drive contributes to total system throughput. Analysis of the effect on performance is discussed in 9.6.2, “Number of drives” on page 202. Capacity requirements are often the only consideration that is used to determine the number of disk drives that are configured in a server. Throughput requirements are usually not well understood or are completely ignored. Good performance by the disk subsystem depends on maximizing the number of read-write heads that can service I/O requests. With RAID (redundant array of independent disks) technology, you can spread the I/O over multiple spindles. There are two options for implementing RAID in a Linux environment: software RAID or hardware RAID. Many System x servers ship with hardware RAID support, but if not, you might want to start with the software RAID options that come with the Linux distributions. Software RAID in the 2.4 Linux kernel distributions is implemented through the md device driver. This driver implementation is device-independent and,
396
Tuning IBM System x Servers for Performance
therefore, is flexible in allowing many types of disk storage such as EIDE or SCSI to be configured as a RAID array. Supported software RAID levels are RAID-0 (striping), RAID-1 (mirroring), and RAID-5 (striping with parity) and can be implemented as part of the initial installation or through the mdadm tool set. It is important to note that the choice of RAID level has a noticeable effect on performance. For more information, see 9.6.1, “RAID strategy” on page 201. If it is necessary to implement a hardware RAID array, you need a RAID controller for your system. In this case, the disk subsystem consists of the physical hard disks and the controller. Tip: In general, adding drives is one of the most effective changes that can be made to improve server performance. For additional, in-depth coverage of the available IBM storage solutions, see Chapter 9, “Disk subsystem” on page 169 as well as the document IBM TotalStorage Disk Solutions for xSeries, SG24-6874, which is available at: http://www.redbooks.ibm.com/abstracts/sg246874.html
12.8.2 Ext3: the default Red Hat file system Since the release of the Red Hat 7.2 distribution, the default file system at the time of installation has been Ext3. This file system is an updated version of the widely used Ext2 file system with the addition of journaling. Highlights of this file system include: Availability: Ext3 always writes data to the disks in a consistent way, so in the case of an unclean shutdown (unexpected power failure or system crash), the server does not have to spend time checking the consistency of the data, thereby reducing system recovery from hours to seconds. Data integrity: By specifying the journaling mode data=journal on the mount command, all data, both file data and metadata, is journaled. Speed: By specifying the journaling mode data=writeback, you can decide on speed versus integrity to meet the needs of your business requirements. Setting the parameter data to writeback will result in a performance increase that is notable in environments where there are heavy synchronous writes. Flexibility: upgrading from existing Ext2 file systems is simple and no reformatting is necessary. By executing the tune2fs command and modifying the /etc/fstab file, you can easily update an Ext2 to an Ext3 file system. Also note that Ext3 file systems can be mounted as ext2 with journaling disabled. Products from many third-party vendors have the capability of manipulating
Chapter 12. Linux
397
Ext3 file systems. For example, PartitionMagic can handle the modification of Ext3 partitions.
12.8.3 ReiserFS: the default SUSE Linux file system The default file system on a SUSE Linux installation since SUSE Linux 7.1 has been ReiserFS, developed by Hans Reiser. From its initial design, key performance aspects have included: Journaling designed into the file system from the beginning to improve reliability and recovery. Faster access through the use of balanced tree data structures that allow for storing both content data and security metadata. Efficient use of disk space because, unlike other file systems, this file system does not rely on block sizes. Note: ReiserFS is not supported by Red Hat Enterprise Linux AS.
12.8.4 File system tuning in the Linux kernel Settings for the default file systems as it is shipped might be adequate for most environments. However, this section discusses a few pointers to help improve overall disk performance.
Accessing time updates The Linux file system keeps records of when files are created, updated, and accessed. Default operations include updating the last-time-read attribute for files during reads and writes to files. Because writing is an expensive operation, eliminating unnecessary I/O can lead to overall improved performance. Mounting file systems with the noatime option prevents the inode access times from being updated. If file update times are not critical to your implementation, as in a Web-serving environment, a user can choose to mount file systems with the noatime flag in the /etc/fstab file as follows: /dev/sdb1 /mountlocation ext3 defaults,noatime 1 2 It is generally a good idea to have a separate /var partition and mount it with the noatime option.
398
Tuning IBM System x Servers for Performance
Tuning the elevator algorithm (kernel 2.4 only) The disk I/O elevator algorithm was introduced as a feature in the V2.4 kernel. It enables the user to tune the algorithm that schedules block I/O by controlling the amount of time an I/O request remains on the queue before being serviced. This is accomplished by adjusting the read and write values of the elevator algorithm. By increasing latency times (that is, larger values for read, write, or both), I/O requests stay on the queue for a longer period of time, giving the I/O scheduler the opportunity to coalesce these requests to perform more efficient I/O and increase throughput. If your Linux server is in an environment with large amounts of disk I/O, finding the right balance between throughput and latency might be beneficial. Linux file systems are implemented as block devices, so improving how often those blocks are read and written can improve file system performance. As a guideline, heavy I/O servers benefit from smaller caches, prompt flushes, and a balanced high-latency read to write. As with other system tuning, tuning the elevator algorithm is an iterative process. You want to baseline current performance, make changes, and then be able to measure the effect of those changes. Example 12-6 shows how to use the /sbin/elvtune command to first show the current settings and then change the values for the read and write queues. Tip: Red Hat’s recommendation is to tune the elevator algorithm so that the read latency (-r) is half the write latency (-w). If any change is made, be sure that the /sbin/elvtune call is added to the /etc/rc.d/rc.local file (Red Hat) or /etc/init.d/boot.local file (SUSE Linux) to make it a persistent change between system boots. Example 12-6 Finding current defaults for your installation and changing them
[root@x232 root]# elvtune /dev/sda /dev/sda elevator ID read_latency: write_latency: max_bomb_segments:
2 2048 8192 6
[root@x232 root]# elvtune -r 1024 -w 2048 /dev/sda /dev/sda elevator ID read_latency: write_latency: max_bomb_segments:
2 1024 2048 6
Chapter 12. Linux
399
If you are using a 2.6 kernel, use the I/O scheduler instead of elvtune. Although elvtune is still available on kernel 2.6 systems, when launching the command under Red Hat, you get the following message: elvtune is only useful on older kernels; for 2.6 use IO scheduler sysfs tunables instead.
The I/O scheduler (kernel 2.6 only) The Linux kernel, the core of the operating system, is responsible for controlling disk access by using kernel I/O scheduling. 2.4 kernel uses a single, robust, general purpose I/O elevator. The I/O schedulers that are provided in Red Hat Enterprise Linux 4 and SUSE Linux Enterprise 9, based on the 2.6 kernel, have advanced the I/O capabilities of Linux significantly. The kernel I/O is selectable at boot time by choosing one of four different I/O schedulers to accommodate different I/O usage patterns. Add the elevator options to the boot loader configuration file (/boot/grub/grub.conf) to select one of the following: The Completely Fair Queuing (CFQ) scheduler, cfq, is the default algorithm in Red Hat Enterprise Linux 4. As the name implies, CFQ maintains a scalable per-process I/O queue and attempts to distribute the available I/O bandwidth equally among all I/O requests. CFQ is well suited for mid-to-large multi-processor systems and for systems which require balanced I/O performance over multiple LUNs and I/O controllers. The Deadline elevator, deadline, uses a deadline algorithm to minimize I/O latency for a given I/O request. The scheduler provides near real-time behavior and uses a round robin policy to attempt to be fair among multiple I/O requests and to avoid process starvation. Using five I/O queues, this scheduler will aggressively re-order requests to improve I/O performance. The NOOP scheduler, noop, is a simple FIFO queue and uses the minimal amount of CPU/instructions per I/O to accomplish the basic merging and sorting functionality to complete the I/O. It assumes performance of the I/O has been or will be optimized at the block device (memory-disk) or with an intelligent HBA or externally attached controller. The Anticipatory elevator, as, introduces a controlled delay before dispatching the I/O to attempt to aggregate and re-order requests improving locality and reducing disk seek operations. This algorithm is intended to optimize systems with small or slow disk subsystems. One artifact of using the as scheduler can be higher I/O latency.
400
Tuning IBM System x Servers for Performance
Selecting the journaling mode of an Ext3 file system Three different journaling options in the Ext3 file system can be set with the data option in the mount command: data=journal This journaling option provides the highest form of data consistency by causing both file data and metadata to be journaled. It also has the higher performance overhead. data=ordered (default) In this mode, only metadata is written. However, file data is guaranteed to be written first. This is the default setting. data=writeback This journaling option provides the fastest access to the data at the expense of data consistency. The data is guaranteed to be consistent as the metadata is still being logged. However, no special handling of actual file data is done and this might lead to old data appearing in files after a system crash. There are three ways to change the journaling mode on a file system: When executing the mount command: mount -o data=writeback /dev/sdb1 /mnt/mountpoint where /dev/sdb1 is the file system that is being mounted. Including it in the options section of the /etc/fstab file: /dev/sdb1 /testfs ext3 defaults,journal=writeback 0 0 If you want to modify the default data=ordered option on the root partition, make the change to the /etc/fstab file listed above, then execute the mkinitrd command to scan the changes in the /etc/fstab file and create a new image. Update grub or lilo to point to the new image. For more information about Ext3, see: http://www.redhat.com/support/wpapers/redhat/ext3/
Tuning ReiserFS Note: ReiserFS is not supported by Red Hat Enterprise Linux AS. One of the strengths of the ReiserFS is its support for a large number of small files. Instead of using the traditional block structure of other Linux file systems, ReiserFS uses a tree structure that has the capability of storing the actual contents of small files or the tails of those that are larger in the access tree itself.
Chapter 12. Linux
401
This file system does not use fixed block sizes, so only the space that is needed to store a file is consumed, leading to less wasted space. There is an option when mounting a ReiserFS file system that improves performance, but at the expense of space. When mounting a ReiserFS, you can disable this tail packing option by specifying notail so that the file system performs a little faster but uses more disk space, as shown in Example 12-7. Example 12-7 Example of mounting a ReiserFS file system with the notail option
/dev/sdb1 /testfs resierfs notail 0 0
Tagged command queuing for SCSI drives Tagged command queuing (TCQ), first introduced in the SCSI-2 standard, is a method by which commands arriving at the SCSI drive are tagged and reordered while in the queue. This implementation can increase I/O performance in server environments that have a heavy, random workload by reordering the requests to optimize the position of the drive head. Recently, this method of queuing and reordering pending I/O requests has been extended to IDE drives and is referred to as ATA TCQ or existing TCQ and Native Command Queuing (NCQ) in the SATA II specification. Some System x servers include the integrated Adaptec AIC-7xxx SCSI controller. By executing cat /proc/scsi/aic7xxx/0, you can check the current TCQ settings in effect. See /usr/src/linux-2.4/drivers/scsi/README.aic7xxx for a detailed description of how to change the default SCSI driver settings. It is not necessary to recompile the kernel to try different settings. You can specify a parameter aic7xxx=global_tag_depth:xx by adding a line in /etc/modules.conf, as shown in Example 12-8. Example 12-8 Setting TCQ option on a server with an Adaptec aic7xxx SCSI card
Edit the /etc/modules.conf file to include options aic7xxx aic7xxx=verbose.global_tag_depth:16 Note: If you make a change to /etc/modules.conf that involves a module in initrd, then it requires a new image through the execution of mkinitrd.
Block sizes The block size, the smallest amount of data that can be read or written to a drive, can have a direct impact on server performance. As a guideline, if your server is handling many small files, then a smaller block size is more efficient. If your server is dedicated to handling large files then a larger block size might improve
402
Tuning IBM System x Servers for Performance
performance. Block sizes cannot be changed on the fly on existing file systems, and only a reformat will modify the current block size. When a hardware RAID solution is being used, careful consideration must be given to the stripe size of the array (or segment in the case of Fibre Channel). The stripe-unit size is the granularity at which data is stored on one drive of the array before subsequent data is stored on the next drive of the array. Selecting the correct stripe size is a matter of understanding the predominant request size performed by a particular application. As a general rule, streaming or sequential content benefits from large stripe sizes by reducing disk head seek time and improving throughput. However, a more random type of activity, such as that found in databases, performs better with a stripe size that is equivalent to the record size. The block sizes that are offered by Linux vary depending on the distribution: Red Hat Enterprise Linux AS with Ext3 allows block sizes of 1 KB, 2 KB, and 4 KB SUSE Linux Enterprise Server with ReiserFS allows a block size of 4 KB only Note: Even though the file system is limited to a maximum block size of 4 KB as listed above, it is still best to set a large stripe size on the RAID controller, because the kernel merges the 4 KB reads and writes into larger requests to the disk subsystem. The maximum size of the request to the drives depends on the driver and the available buffer memory. We have seen better performance with a larger stripe size on the disk due to this request merging.
Guidelines for setting up partitions A partition is a contiguous set of blocks on a drive that are treated as though they were independent disks. The default Linux installation creates a very monolithic install with only three partitions: A swap partition (automatically set to 2x RAM or 2 GB, whichever is larger) A small boot partition, /boot (for example, 100 MB) All remaining space dedicated to / There is a great deal of debate in Linux circles about the optimal disk partition. A single root partition method can lead to problems in the future if you decide to redefine the partitions because of new or updated requirements. Alternatively, too many partitions can lead to a file system management problem. During the installation process, Linux distributions allow you to create a multi-partition layout.
Chapter 12. Linux
403
There are benefits to running Linux on a multi-partitioned disk: Improved security with finer granularity on file system attributes For example, the /var and /tmp partitions are created with attributes that permit very easy access for all users and processes on the system and are susceptible to malicious access. By isolating these partitions to separate disks, you can reduce the impact on system availability if these partitions need to be rebuilt or recovered. Improved data integrity, because loss of data with a disk crash would be isolated to the affected partition For example, if there is no RAID implementation on the system (software or hardware) and the server suffers a disk crash, only those partitions on that bad disk would have to be repaired or recovered. New installation and upgrades can be done without affecting other more static partitions For example, if the /home file system has not been separated to another partition, it is overwritten during an operating system upgrade and all user files that are stored on it are lost. More efficient backup process Partition layouts must be designed with backup tools in mind. It is important to understand whether backup tools operate on partition boundaries or on a more granular level like file systems. Table 12-7 lists some of the partitions that you might want to consider separating out from the root directory to provide more flexibility and better performance in your environment. Table 12-7 Linux partitions and server environments
Partition
Contents and possible server environments
/home
A file server environment benefits from separating out /home to its own partition. This is the home directory for all users on the system if there are no disk quotas implemented, so separating this directory should isolate a user’s runaway consumption of disk space.
/tmp
If you are running a high-performance computing environment, large amounts of temporary space are needed during compute time, then are released upon completion.
/usr
This is the directory where the kernel source tree and Linux documentation (as well as most executable binaries) are located. The /usr/local directory stores the executables that need to be accessed by all users on the system and is a good location to store custom scripts that are developed for your environment. If it is separated to its own partition, then files will not need to be reinstalled during an upgrade.
404
Tuning IBM System x Servers for Performance
Partition
Contents and possible server environments
/var
The /var partition is important in mail, Web, and print server environments because it includes the log files for these environments as well as the overall system log. Chronic messages can flood and fill this partition. If this occurs and the partition is not separate from the root directory, service interruptions are possible. Depending on the environment, further separation of this partition is possible by separating out /var/spool/mail for a mail server or /var/log for system logs.
/opt
The installation of some third-party software products, such as Oracle’s database server, default to this partition. If not separate, the installation will continue under / and, if there is not enough space allocated, might fail.
For a more detailed and in-depth understanding of how Linux distributions handle file system standards, see the File System Hierarchy project’s home page: http://www.pathname.com/fhs
12.8.5 The swap partition The swap device is used when physical RAM is fully in use and the system needs additional memory. When there is no free memory available on the system, it begins paging the least-used data from memory to the swap areas on the disks. The initial swap partition is created during the Linux installation process with current guidelines stating that the size of the swap partition should be two times physical RAM. The maximum total size of swap that is supported is 64 GB for both kernel 2.4 and kernel 2.6. If you add more memory to the server after the initial installation, you must configure additional swap space. There are two ways to configure additional swap after the initial install: You can create a free partition on the disk as a swap partition, which can be difficult if the disk subsystem has no free space available. In that case, you can create a swap file. If there is a choice, the preferred option is to create additional swap partitions. There is a performance benefit because I/O to the swap partitions bypasses the file system and all of the overhead involved in writing to a file.
Chapter 12. Linux
405
Another way to improve the performance of swap partitions or files is to create multiple swap areas. Linux can take advantage of multiple swap partitions or files and perform the reads and writes in parallel to the disks. After creating the additional swap partitions or files, the /etc/fstab file includes entries such as those shown in Example 12-9. Example 12-9 /etc/fstab file
/dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
swap swap swap swap
swap swap swap swap
sw sw sw sw
0 0 0 0
0 0 0 0
Under normal circumstances, Linux would use the /dev/sda2 swap partition first, then /dev/sdb2, and so on, until it had allocated enough swapping space. This means that perhaps only the first partition, /dev/sda2, would be used if there is no need for a large swap space. The maximum supported number of swapfiles is 32. Spreading the data over all available swap partitions improves performance because all read/write requests are performed simultaneously to all selected partitions. If you change the file as shown in Example 12-10, you assign a higher priority level to the first three partitions. Example 12-10 Modified /etc/fstab to make parallel swap partitions
/dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
swap swap swap swap
swap swap swap swap
sw,pri=3 sw,pri=3 sw,pri=3 sw,pri=1
0 0 0 0
0 0 0 0
Swap partitions are used from the highest priority to the lowest (where 32767 is the highest and 0 is the lowest). Giving the same priority to the first three disks causes the data to be written to all three disks; the system does not wait until the first swap partition is full before it starts to write on the next partition. The system uses the first three partitions in parallel and performance generally improves. The fourth partition is used if additional space is needed for swapping after the first three are completely filled up. It is also possible to give all partitions the same priority to stripe the data over all partitions, but if one drive is slower than the others, performance will decrease. A general rule is that the swap partitions should be on the fastest drives available. Note: The swap space is not a replacement for RAM because it is stored on physical drives that have a significantly slower access time than memory.
406
Tuning IBM System x Servers for Performance
12.9 Tuning the network subsystem The network subsystem should be tuned when the operating system is first installed as well as when there is a perceived bottleneck in the network subsystem. An issue here can affect other subsystems. For example, CPU utilization can be affected significantly, especially when block sizes are too small, and memory use can increase if there is an excessive number of TCP connections.
12.9.1 Preventing a decrease in performance The following sysctl commands are used primarily to change security settings, but they also have the side effect of preventing a decrease in network performance. These commands are changes to the default values. Disabling the following parameters prevents a hacker from using a spoofing attack against the IP address of the server: sysctl sysctl sysctl sysctl
-w -w -w -w
net.ipv4.conf.eth0.accept_source_route=0 net.ipv4.conf.lo.accept_source_route=0 net.ipv4.conf.default.accept_source_route=0 net.ipv4.conf.all.accept_source_route=0
(Red Hat Enterprise Linux AS only) This command enables TCP SYN cookies, which protect the server from syn-flood attacks, both denial-of-service (DoS) or distributed denial-of-service (DDoS): sysctl -w net.ipv4.tcp_syncookies=1 Note: This command is valid only when the kernel is compiled with CONFIG_SYNCOOKIES. These commands configure the server to ignore redirects from machines that are listed as gateways. Redirect can be used to perform attacks, so we only want to allow them from trusted sources: sysctl sysctl sysctl sysctl
-w -w -w -w
net.ipv4.conf.eth0.secure_redirects=1 net.ipv4.conf.lo.secure_redirects=1 net.ipv4.conf.default.secure_redirects=1 net.ipv4.conf.all.secure_redirects=1
In addition, you could allow the interface to accept or not accept any ICMP redirects. The ICMP redirect is a mechanism for routers to convey routing information to hosts. For example, the gateway can send a redirect message to a host when the gateway receives an Internet datagram from a host on a network to which the gateway is attached. The gateway checks the routing table to get the address of the next gateway, and the second gateway routes
Chapter 12. Linux
407
the datagram’s Internet to destination on network. Disable these redirects using the following commands: sysctl sysctl sysctl sysctl
-w -w -w -w
net.ipv4.conf.eth0.accept_redirects=0 net.ipv4.conf.lo.accept_redirects=0 net.ipv4.conf.default.accept_redirects=0 net.ipv4.conf.all.accept_redirects=0
If this server does not act as a router, then it does not need to send redirects, so they can be disabled using the following commands: sysctl sysctl sysctl sysctl
-w -w -w -w
net.ipv4.conf.eth0.send_redirects=0 net.ipv4.conf.lo.send_redirects=0 net.ipv4.conf.default.send_redirects=0 net.ipv4.conf.all.send_redirects=0
Configure the server to ignore broadcast pings or smurf attacks: sysctl -w net.ipv4.icmp_echo_ignore_broadcasts=1 Ignore all kinds of icmp packets or pings: sysctl -w net.ipv4.icmp_echo_ignore_all=1 Some routers send invalid responses to broadcast frames, and each one generates a warning that is logged by the kernel. These responses can be ignored using this command: sysctl -w net.ipv4.icmp_ignore_bogus_error_responses=1
12.9.2 Tuning in TCP and UDP You can use the following commands for tuning servers that support a large number of multiple connections: For servers that receive many connections at the same time, the TIME-WAIT sockets for new connections can be reused. This command is useful in Web servers, for example: sysctl -w net.ipv4.tcp_tw_reuse=1 If you enable this command, you should also enable fast recycling of TIME-WAIT sockets status as follows: sysctl -w net.ipv4.tcp_tw_recycle=1
408
Tuning IBM System x Servers for Performance
Figure 12-6 shows that with these parameters enabled, the number of connections is reduced significantly. This reduction is good for performance because each TCP transaction maintains a cache of protocol information about each of the remote clients. In this cache, information such as round-trip time, maximum segment size, and congestion window are stored. For more details, review RFC 1644 from: http://www.ietf.org/rfc/rfc1644.txt
tcp_tw_reuse and tcp_tw_recycle enabled.
tcp_tw_reuse and tcp_tw_recycle disabled.
With both tcp_tw_reuse and tcp_tw_recycle enabled, the information about the hosts does not have to be obtained again and the TCP transaction can start immediately, preventing the unnecessary traffic.
Figure 12-6 Parameters reuse and recycle enabled (left) and disabled (right)
The parameter tcp_fin_timeout is the time to hold a socket in state FIN-WAIT-2 when the socket is closed at the server. A TCP connection begins with a three-segment synchronization SYN sequence and ends with a three-segment FIN sequence, neither of which holds data. By changing the tcp_fin_timeout value, the time from the FIN sequence to when the memory can be freed for new connections can be reduced, thereby improving performance. You should change this value,
Chapter 12. Linux
409
however, only after careful monitoring, because there is a risk of overflowing memory due to the number of dead sockets. sysctl -w net.ipv4.tcp_fin_timeout=30 One of the issues found in servers with many simultaneous TCP connections is the large number of connections that are open but unused. TCP has a keepalive function that probes these connections and, by default, drops them after 7200 seconds (2 hours). This length of time might be too large for your server and can result in excess memory usage and a decrease in server performance. Setting keepalive to 1800 seconds (30 minutes), for example, might be more appropriate: sysctl -w net.ipv4.tcp_keepalive_time=1800 Set the max operating system send buffer size (wmem) and receive buffer size (rmem) to 8 MB for queues on all protocols as follows: sysctl -w net.core.wmem_max=8388608 sysctl -w net.core.rmem_max=8388608 These commands specify the amount of memory that is allocated for each TCP socket when it is created. In addition, you should also use the following commands for send and receive buffers. They specify three values: minimum size, initial size, and maximum size: sysctl -w net.ipv4.tcp_rmem="4096 87380 8388608" sysctl -w net.ipv4.tcp_wmem="4096 87380 8388608" The third value must be the same as or less than the value of wmem_max and rmem_max. (SUSE Linux Enterprise Server only) Validate the source packets by reserved path. By default, routers route everything, even packets that obviously are not meant for this network. These packets can be dropped, by enabling the appropriate filter: sysctl sysctl sysctl sysctl
-w -w -w -w
net.ipv4.conf.eth0.rp_filter=1 net.ipv4.conf.lo.rp_filter=1 net.ipv4.conf.default.rp_filter=1 net.ipv4.conf.all.rp_filter=1
When the server is heavily loaded or has many clients with bad connections with high latency, it can result in an increase in half-open connections. This is very common for Web servers, especially when there are many dial-up users.
410
Tuning IBM System x Servers for Performance
These half-open connections are stored in the backlog connections queue. You should set tcp_max_syn_backlog to at least 4096 (the default is 1024). Setting this value is useful even if your server does not receive this kind of connection, because it can still be protected from a denial-of-service (syn-flood) attack. sysctl -w net.ipv4.tcp_max_syn_backlog=4096 We should set the ipfrag parameters particularly for NFS and Samba servers. Here, we can set the maximum and minimum memory used to reassemble IP fragments. When the value of ipfrag_high_thresh in bytes of memory is allocated for this purpose, the fragment handler drops packets until ipfrag_low_thres is reached. Fragmentation occurs when there is an error during the transmission of TCP packets. Valid packets are stored in memory (as defined with these parameters) while corrupted packets are retransmitted. For example, to set the range of available memory to between 256 MB and 384 MB use the following commands: sysctl -w net.ipv4.ipfrag_low_thresh=262144 sysctl -w net.ipv4.ipfrag_high_thresh=393216
12.10 SUSE Linux Enterprise Server 10 At the time of writing, Novell announced a new release of SUSE Linux Enterprise Server—SLES 10. This section provides a general overview of this product. Note: SLES 10 provides the features that we discuss in this section. Some of these features, however, are implemented previously in SLES 9 or RHEL 4.
12.10.1 Virtualization SLES 10 includes the following features: Xen 3.0 Virtual Machine Monitor (VMM) Runs many smaller virtual machines (VMs) on a single server, each with a separate operating system and application instance. Processor architectures Provide support for SMP x86 and SMP x86-64 systems. These architectures also provide virtual SMP support for virtual servers, virtual block-device support, and virtual network-interface support.
Chapter 12. Linux
411
Native Hardware Support Fully supports AMD Pacifica and Intel VT hardware. Administration interface Creates virtual machine profiles and configures individual virtual servers with a fully integrated YaST module. Guest operating systems Run multiple SLES 10 guest servers on the same SLES 10 host server.
12.10.2 Administration and manageability SLES 10 includes the following features: YaST2 Covers a wide range of management tasks and has been enhanced to give a consistent management experience across all SUSE Linux platforms. CIM management Delivers an open Web-based Enterprise Management (WBEM) Common Information Model Object Manager (CIMOM) as well as a variety of CIM providers for consumption by management frameworks. LiMAL management library Provides a common operating system interface for use by management utilities such as YaST, CIM, and third-party tools. Directory Integration — Supported directories Give all organizations-both small and large-a choice of LDAP-compliant directory services: – Microsoft Active Directory – OpenLDAP – Novell eDirectory SPident Queries the RPM database and matches each installed package against all known service packs (SPs). Only the newest SP, the one that does not include any out-of-date packages, is marked as being installed. Intelligent Platform Management Interface (IPMI) 1.4.19 Uses IPMItool to monitor health, inventory, and remote power control of OpenIPMI 1.4.19 compatible servers.
412
Tuning IBM System x Servers for Performance
iprutils 2.1.2 Includes a suite of utilities to manage and configure small computer system interface (SCSI) devices supported by the ipr SCSI storage device driver. Version 2.1.2 includes utilities for the IBM Power Linux RAID adapters. net-snmp 5.3.0.1 Provides tools and libraries relating to Simple Network Management Protocol (SNMP). These resources include an extensible agent, an SNMP library, tools to request or set information from SNMP agents, and tools to generate and to handle SNMP traps.
12.10.3 Security SLES 10 includes the following features: Novell AppArmor 2.0 Provides easy-to-use Linux security software that protects Linux servers and applications from malicious threats. Novell AppArmor features: – YaST-integrated administration tools for configuration, maintenance and automated development of per-program security policy – Pre-defined security policies for standard Linux programs and services – Robust reporting and alerting capabilities to facilitate regulatory compliance The following features are new in Novell AppArmor 2.0: – CIM-enabled clients that integrate with industry-standard management consoles – ZENworks Linux Management integration for profile distribution and report aggregation – Enhanced auditing and reporting MIT Kerberos 5 release 1.4.3 Uses secret-key cryptography to provide strong authentication for client/server applications. Snort 2.4.3 Provides a lightweight network-intrusion detection system to enhance overall security. Snort performs protocol analysis and content searching and matching. It can also be used to detect a variety of attacks and probes, such as buffer overflows, stealth port scans, CGI attacks, SMB probes, OS fingerprinting attempts and much more.
Chapter 12. Linux
413
Advanced Intrusion Detection Environment (AIDE) 0.10 Enhances security by performing data-integrity assurance with the file manipulation monitoring system.
12.10.4 Scalability and performance SLES 10 includes the following features: CPU scalability Scales to 512 processors on most standard architectures and up to 1024 processors on IA-64 systems. CPU performance and scheduler enhancements Provide hyper-threading, which enables multi-threaded server software applications to execute threads in parallel within each individual server processor. CPUSET Delivers lightweight kernel objects (CPUSETs) that enable users to partition their multiprocessor machines in terms of CPUs. CPUSETs are strong "jails," meaning that processes running on predefined processors will not be able to run on other processors. Pluggable I/O scheduler Improves performance and allows administrators to tune the server to match its usage with four I/O behavior policies: – Complete Fair Queuing (CFQ): This default scheduler is suitable for a wide variety of applications, especially desktop and multimedia workloads. – Deadline: The deadline I/O scheduler implements a per-request service deadline to ensure that no requests are neglected. Deadline policy is best for disk-intensive database applications. – Anticipatory: The anticipatory I/O scheduler uses the deadline mechanism plus an anticipation heuristic to predict the actions of applications. The anticipation heuristic is best suited for file servers. – No-Op: This no-operation mode does no sorting at all and is used only for disks that perform their own scheduling or that are randomly accessible. See “The I/O scheduler (kernel 2.6 only)” on page 400 for more information. Raw I/O Improves database performance by reducing read/write times, which is especially useful for SCSI and Fibre Channel devices that transfer data directly to a buffer in the application address space. Raw I/O utilizes
414
Tuning IBM System x Servers for Performance
high-bandwidth and low-overhead SCSI disk I/O by skipping kernel buffering and I/O queuing. Intel I/O Acceleration Technology Provides a kernel that supports the I/OAT networking acceleration technology from Intel. Customers using Intel server chipsets will experience increased network performance. Memory limits Offer the following scalability: – 10 TB in production for 64-bit architectures – Large memory support on 32-bit machines through efficient memory management (anon-vma and objrmap) – Up to 1 GB Highmem support on x86 systems – Up to 4 GB with Physical Address Extensions (PAE) support on x86 systems
12.10.5 Storage and high availability SLES 10 includes the following features: Network File System (NFS) V4 Delivers network file-sharing capabilities for UNIX and Linux installations. SUSE Linux Enterprise 10 supports NFS versions 2 and 3 and also 4 over both UDP and TCP. Version 4 includes performance improvements, mandates strong security and introduces a stateful protocol. Oracle Cluster File System 2 (OCFS2) Runs in shared Oracle home installations and makes the management of Oracle Real Application Cluster (RAC) installations easier. The following features are new in OCFS2: – Node and architecture local files that use Context Dependent Symbolic Links (CDSL) – Support on all architectures – Network-based pluggable Distributed Lock Manager (DLM) – Improved journaling and node recovery using the Linux Kernel Java Debugger (JDB) subsystem – Improved performance of metadata operations – Improved data caching and locking
Chapter 12. Linux
415
Heartbeat 2.0.3 Provides core cluster membership and messaging infrastructure. It implements the Open Clustering Framework APIs and provides services for node fencing, fault isolation and health monitoring. The following features are new in Heartbeat 2.0.3: – YaST enhancement to configure more than two-node failover – Sub-second failure detection – I/O data integrity checks that are performed before resources are moved to alternate nodes – Failed nodes return to action set to automatic or manual Multipath I/O Enables greater load balancing and fault tolerance by accessing storage devices simultaneously through multiple channels. Multipath I/O tools include: – multipath: scans the system for multipathed devices, assembles them, and updates the device-mapper's maps – multipathd: waits for maps events and then executes multipath – devmap-name: provides a meaningful device name to udev for devmaps – kpartx: maps linear devmaps to device partitions – Version 0.4.6 improves support for EMC storage arrays. Serviceability—Linux Kernel Crash Dump (Kexec/Kdump) Provides a new and preferred way to perform kernel crash dumps. Kexec/Kdump is supported on i386™, x86-64, and ppc/ppc64 systems. High Availability Storage Foundation Integrates Oracle Cluster File System 2 (OCFS2), Enterprise Volume Management System (EVMS) and Heartbeat 2 clustering services to deliver an entirely open source high-availability (HA) storage solution. Enterprise Volume Management System (EVMS) Administers storage through a single mechanism. Administrators can use EVMS to manage RAID, LVM, various file-system formats, disk checking and maintenance, bad block relocation and more. Distributed Replicated Block Device 0.7.15 (DRBD) Builds single partitions from multiple disks that mirror each other. Using this disk-management tool, performance is similar to a RAID1 system, but it runs over a network. DRBD 0.7.15 allows partition sizes to be changed at runtime.
416
Tuning IBM System x Servers for Performance
iSCSI Links data storage facilities through local area networks (LANs), wide area networks (WANs), or the Internet. SUSE Linux Enterprise Server can act as both a target and initiator. For example, SUSE Linux Enterprise Server can be used as a SAN box (target) or as an iSCSI client (initiator).
12.10.6 Server services SLES 10 includes the following features: File and Print Services—Samba 3.0.21b Provides authentication, file, print, and WINS services for Microsoft Windows Client systems. Additionally, Samba allows a Linux client to integrate into existing Microsoft domains. With Samba 3.0.21b, Linux clients can also join Active Directory domains and authenticate against Active Directory servers. Mail Services—Postfix 2.1.1 and Sendmail 8.12.10 Provide mail serving. Postfix 2.1.1 is the default mail server for SUSE Linux products. Sendmail 8.12.10 is available as an alternative and for backward compatibility. IMAP, specifically Cyrus IMAP Daemon 2.2 Enhances data protection and generally runs on sealed servers. MySQL 5.0 Provides the popular open source database, now with rollback, crash recovery, low-level locking, database replication, clustering, full-text indexing, and searching. PostgreSQL 8.1 Offers another flexible and extensible open source database. The following features are new in PostgreSQL 8.1: – Database roles that simplify the management of large numbers of users – IN/OUT parameters that improve support of complex business logic for J2EE™ and .NET applications – Improved Two-Phase Commit to support WAN applications, heterogeneous data centers and ACID-compliant transactions – Improved multiprocessor (SMP) performance – 64-bit shared memory to support up to 2 TB of RAM on 64-bit platforms
Chapter 12. Linux
417
Apache Web server 2.2.0 Deserves its reputation as the number-one HTTP server on the Internet. Apache 2.2.0 features a hybrid multi-process/multi-threaded implementation. It also supports extension modules for IPv6, filtering, multi-language error responses, simplified configuration and a new API. Geronimo Delivers an open source Java Application Server from apache.org. Key features in Geronimo include: – J2EE 1.4 compatibility – New ASF code for a complete J2EE stack – Full Tomcat integration Java 1.4 Provides the popular object-oriented development language. SUSE Linux Enterprise 10 ships with the Java 2 platform, version 1.4.2. PHP 5.1 Provides a general-purpose scripting language that is especially suited for Web development. Key features in PHP 5.1 include: – Completely rewritten date-handling code, with improved time zone support – Significant performance improvements (compared to PHP 5.0.X) – Default enablement of the PHP Data Objects (PDO) extension – More than 30 new functions in various extensions and built-in functionality – Bundled libraries, PCRE and SQLite upgraded to latest versions – PEAR upgraded to version 1.4.5 Python Delivers an object-oriented interpreted language that is often used for rapid development of cross-platform applications. SUSE Linux Enterprise 10 includes the current Python version along with bindings for QT, Gtk, LDAP, XML, MySQL, Tk, and curses. Tcl/Tk scripting tools Enable the rapid development of cross-platform GUI applications. Ruby Provides an interpreted scripting language designed for quick and easy object-oriented programming. The latest version is suitable for many of the same processing tasks performed by Python or Perl.
418
Tuning IBM System x Servers for Performance
Shell scripting Delivers bash (default), ksh, tcsh, and zsh in SUSE Linux Enterprise 10. Orarun Significantly simplifies Oracle configuration. With orarun, you can: – Create Oracle users and groups – Set Oracle environment variables – Set Oracle-recommended kernel parameters – Automate the start and stop of Oracle system components – Rely on YaST integration Suse-sapinit Automatically or manually start and stop SAP-application components at system startup, shutdown, or reboot.
12.10.7 Application and developer services SLES 10 includes the following features: Mono 1.2 Accommodates developers who prefer to use .NET development skills. SUSE Linux Enterprise Server is the only distribution to include Mono. Mono 1.2 includes the following features: – – – – – –
Optimized C# 2.0 compiler Gtk# 2.0 Mono Debugger System.Windows.Forms implementations VB.NET runtime (VB.NET compiler preview) db4o for object persistence
Developer tool-chain Delivers the latest developer tool-chain with the following features: – – – – –
binutils 2.16.91 GCC 4.1.0 glibc 2.3.90 GDB 6.4 Tools for C, C++, Fortran77, Java, Ada and Objective-C
NUMA development tools Fine-tune applications for NUMA usage on both x86-84 (Opteron) and IA-64 (Itanium) systems.
Chapter 12. Linux
419
12.11 Xen virtualization Virtualization has become a key requirement for the enterprise and results from a need to focus on reduced total cost of ownership (TCO) for enterprise computing infrastructure. Most servers today run at less than 15% utilization, meaning that most server capacity is wasted. Operating system virtualization allows multiple operating system and application images to share each server. Because every physical server can host multiple virtual servers, the number of servers is reduced. However, today’s virtualization offerings are also facing performance issues. To bypass these issues, a low level virtualization software layer—a hypervisor—has been introduced. XenSource3 founders created the Xen open source hypervisor, which is now developed collaboratively by over 20 major enterprises working on the Xen project. Xen hypervisor fully supports VT-x hardware virtualization from Intel and offers a software layer to this facility. Pacifica from AMD is not yet supported.
12.11.1 What virtualization enables Operating system virtualization is achieved by inserting a layer of software between the operating system and the underlying server hardware. This layer is responsible for allowing multiple operating system images (and their running applications) to share the resources of a single server. In this environment, each operating system believes that it has the resources of the entire machine under its control, but the virtualization layer, or hypervisor, transparently ensures that resources are properly shared between different operating images and their applications (Figure 12-7).
3
420
Portions of the material in this section are from XenSource, reprinted by permission.
Tuning IBM System x Servers for Performance
Guest OS and API
Guest OS and API
Guest OS and API
Xen Hypervisor / virtualization layer
Virtualized hardware
Server with Intel VT or AMD Pacific
Phyical hardware
Figure 12-7 Virtualization concept
In operating system virtualization, the hypervisor must manage all hardware structures to ensure that each operating system, when running, has a consistent view of the underlying hardware. Although there are several methods to manage hardware structures, the simplest method (with the worst performance) is to provide a software emulation layer of the underlying chipset. This method imposes severe performance overheads, restricts the technique to a single chipset architecture (such as x86), and involves patching the kernel of a running operating system dynamically to prevent it from modifying the state of the hardware. The additional overhead that is required to manage the hardware state for the operating system and to present to it an emulated chipset abstraction causes a significant performance overhead—frequently as much as 30% to 50% of the overall system resources.
12.11.2 Full virtualization versus paravirtualization Xen is a virtualization product that uses paravirtualization. Full virtualization is the concept of creating a virtual layer, typically a hypervisor, that fully simulates a standard x86 system. In this environment, the guest operating system does not need to be modified to be aware of the virtualization layer and can run natively on the virtualization layer as thought it were on a standard x86 system.
Chapter 12. Linux
421
Both VMware ESX Server and Microsoft Virtual Server implement a full virtualization technology, as shown in Figure 12-8.
(Standard) guest OS Standard x86 hardware calls
Full x86 hardware virtualized interface Hypervisor
x86 hardware
Figure 12-8 Full virtualization architecture
With paravirtualization, the guest operating system is modified to be virtualization-aware so that the it can call the hypervisor directly to perform low-level functions, as illustrated in Figure 12-9.
(Paravirtualized) guest OS Specific virtual-aware hypervisor calls Standard x86 hardware calls
Full x86 hardware virtualized interface Hypervisor
x86 hardware
Figure 12-9 Paravirtualization architecture
422
Tuning IBM System x Servers for Performance
There are at least two reasons for doing paravirtualization: Reduced complexity of the virtualization software layer Because x86 historically does not support virtualization, the full virtualization approach must implement mechanisms for which it traps certain privileged guest operating system calls and then translate them into instructions that suit the virtualized environment. This process is accomplished using a technique called binary translation. With this process, some current virtualization products trap and translate certain instructions that are directed at the hardware, thereby allowing guest operating systems to operate as though they were in full control of the hardware. This is where paravirtualization differs from full virtualization. Instead of the hypervisor trapping low-level instructions and transforming those, it is the virtual-aware guest operating system that behaves differently and becomes aware that it is not the only operating system on the server. This in turn means that the hypervisor does not need to provide the complexity of entirely simulating an x86 computer. The result is that the entire system can be streamlined to run more efficiently. Performance The second reason to implement paravirtualization is performance. The full virtualization approach often suffers in terms of performance because there are many overheads in running a standard, unmodified guest operating system on a virtualization layer. For example, a standard unmodified guest operating system typically checks everything before passing the information to the hardware (CPU, memory, and I/O). So does the virtualization layer (which it needs to do because it is closest to the hardware).
12.11.3 CPU and memory virtualization In Xen paravirtualization, virtualization of CPU, memory, and low level hardware interrupts are provided by a low level, hypervisor layer. When the operating system updates hardware data structures, it collaborates with the hypervisor by making calls into an API that is provided by the hypervisor. This collaboration allows the hypervisor to keep track of all the changes that the operating system makes and to decide optimally how to manage the state of hardware data structures on context switches. The hypervisor is mapped into the address space of each guest operating system so that there is no context switch overhead between any operating system and the hypervisor. Finally, by co-operatively working with the guest operating systems, the hypervisor determines the I/O requirements of the guest operating system and can make the operating system aware that it is being virtualized.
Chapter 12. Linux
423
12.11.4 I/O virtualization Paravirtualization provides significant benefits in terms of I/O virtualization. In the Xen product, I/O is virtualized using only a single set of drivers for the entire system (across all guests and the hypervisor), unlike emulated virtualization in which each guest has its own drivers and the hypervisor has yet another set of drivers. In each Xen hypervisor guest, simple paravirtualizing device drivers replace hardware-specific drivers for the physical platform. Paravirtualizing drivers are independent of all physical hardware, but represent each type of device (for example, block I/O, Ethernet, and USB). Moreover, in the Xen architecture the drivers run outside the base hypervisor, at a lower level of protection than the core of the hypervisor itself. In this way the hypervisor can be protected from bugs and crashes in device drivers and can make use of any device drivers that are available on the market. Also, the virtualized operating system image is much more portable across hardware, because the low levels of the driver and hardware management are modules that run under control of the hypervisor. For more information about Xen, visit: http://www.xensource.com/products/xen/index.html
424
Tuning IBM System x Servers for Performance
13
Chapter 13.
VMware ESX Server VMware ESX Server1 is currently the most popular virtualization software product for the Intel processor-based server market. Compared to hosted virtualization solutions such as VMware Server and Microsoft Virtual Server, ESX Server offers the advantage of eliminating one layer of overhead, namely, the host operating system. Because the ESX Server kernel runs directly on the hardware by incorporating a hypervisor virtualization solution, the system has improved performance and stability. ESX Server is also more suitable for enterprise-level solutions because it features important redundancy features such as multi-pathing and link aggregation. ESX Server is capable of simultaneously hosting many different operating systems and applications. With the support for tools—such as P2V (a physical-to-virtual-machine migration utility), VMotion, and Virtual Center — the ESX Server system is truly built for enterprise deployments. You can obtain additional information about all aspects of Virtualization from Virtualization on the IBM System x3950 Server, SG24-7190: http://www.redbooks.ibm.com/abstracts/sg247190.html
1
Portions of the material in this section are from VMware, Inc., reprinted by permission.
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
425
13.1 Introduction Performance tuning for large ESX Server systems can be a challenging task. ESX Server can be a heavy load on your server hardware, and depending on the workload and the number of virtual machines, your server might experience a bottleneck in one of the server’s subsystems. It is, therefore, of paramount importance that you design and configure the hardware so that you will not run into a bottleneck that could limit the system performance. It is also important to understand that ESX Server only virtualizes your workload which means in fact that you will have to size your ESX Server system according to the planned usage. Virtualizing your infrastructure servers or your terminal servers will have a large impact on the way you size and configure your system. It is also important to remember that ESX Server virtualizes your hardware. You will also have to implement the specific tuning options we discussed for the respective operating systems and applications in each of the guest virtual machines.
13.1.1 Understanding VMware performance concepts Before discussing some of the tuning options ESX Server provides, it is important to understand the impact of virtualization on performance. ESX Server virtualizes the hardware and provides an environment for multiple operating systems to run on one physical machine. By default, every virtual machine has equal rights when it comes to the access of hardware such as disk or network. You should, therefore, not expect to see one virtual machine performing as well as a single operating system on the very same hardware because ESX Server imposes limits on every virtual machine. For example, consider a server that is attached to a SAN that delivers about 250 MBps throughput. If you install ESX Server on the same server, create a virtual machine and re-run the test, throughput could be as little as 25 MBps to 50 MBps. Is ESX Server performing poorly? Now, create another virtual machine and run the test in parallel in both virtual machines. More than likely both VMs will see throughput of 25 MBps to 50 MBps. You could do this until a bottleneck appears either in the ESX Server kernel or the SAN, which is, in this example, more likely. From this example, you can see that ESX Server is made for parallel scalability rather than for peak performance of a single virtual machine. If you have workload that really requires peak performance of any subsystem you would not want to put this application on a ESX Server environment. However, if you have a number of servers that do not require significant amounts of I/O and CPU
426
Tuning IBM System x Servers for Performance
performance (and that is usually where virtualization comes into play), going to ESX Server will save you hardware and the associated operating costs. Some server applications do perform close to the native level — that is, on non-virtualized hardware. Examples include ones which have mostly user-level application code. The remainder of this chapter is divided into the following sections: 13.2, “General ESX Server tuning considerations” on page 427 13.3, “ESX Server 2.5 and later features” on page 443 13.4, “ESX Server 3.0 features” on page 459
13.2 General ESX Server tuning considerations The configurations that we discuss in this section are applicable to both ESX Server 2.0 and later as well as ESX Server 3.0.
13.2.1 Hardware layout When configuring ESX Server on a System x server, you have a very broad range of products to choose from. You can scale from a simple 2-socket system with internal disks to a 32-socket single core x445 with 64 GB of RAM and SAN attachment. Therefore it is impossible to give you a best of solution—again, the hardware configuration will depend on your organization’s needs. Table 13-1 shows the maximum CPU configurations for ESX Server. Table 13-1 Maximum CPU configurations with ESX Server (nodes refers to the number of x3950 servers)
ESX Server Version
Single-core CPUs, HT disabled
Single-core CPUs, HT enabled
Dual-core CPUs, HT disabled
Dual-core CPUs, HT enabled
2.5.x
4 nodes, 16 CPUs
2 nodes, 8 CPUs
2 nodes, 8 CPUs
1 node, 4 CPUs
3.0
8 nodes, 32 CPUs
4 nodes, 16 CPUs
4 nodes, 16 CPUs
2 nodes, 8 CPUs
Note: Even though the configuration that will allow you to make use of the maximum number of the 32 logical processors available in ESX 3.0 is a 2-node, 8 dual-core CPUs with Hyper-Threading enabled (2-node x 4 CPUs x 2 cores x 2 logical Hyper-Threading-enabled processors = 32), an 8-node, 32 single core CPUs configuration would still offer the highest performance.
Chapter 13. VMware ESX Server
427
For ESX Server, the hardware subsystems that are most likely to be bottlenecks are: Memory Disk I/O Network bandwidth The typical applications that you would run in a virtualized environment such as ESX Server do not cause a CPU bottleneck. To avoid bottlenecks to the memory subsystems, always try to go with the systems that have the most available fastest front-side bus bandwidth. The x3950 for example, incorporates a snoop filter which increases the available front side bus bandwidth. ESX Server performs a lot of CPU-to-memory and I/O-to-memory operations, and they all go through the front-side bus. Disabling prefetching in BIOS might also provide additional front-side bus bandwidth, if that option is available. Also, try to configure as much memory as possible (and as affordable) to prevent the VMware kernel from swapping. Note that the amount of physical memory required is dependent on the memory requirements of the applications that are to be run inside the VMs. Just as important as the proper memory configuration is the disk subsystem. Disk subsystems often limit performance even on simple non-virtualized systems, and disks have an even greater significance in a ESX Server environment. The disks that hold the ESX Server kernel, the core dump, and the Console OS files should if possible be a dedicated RAID-1 array residing on direct-attached storage. However, in some circumstances, it might become necessary to boot from a SAN, iSCSI, or boot from a single hard drive. We also recommend that you not use the onboard LSI controller found on some servers but rather go with an external RAID controller such as a ServeRAID Adapter. In many cases the onboard LSI RAID controllers have a slower CPU, no cache, and no battery backup, so they are not suited to production level ESX Server systems.
428
Tuning IBM System x Servers for Performance
For the VMFS (Virtual Machine File System) storage (the storage on which your virtual machines will reside), we recommend you configure the fastest disk subsystem available for your budget. Some key aspects of your design should be: Use a RAID array of many disks. The more disks, the better performance. Use 10 K or 15 K RPM disks. If using SCSI, use Ultra320 SCSI drives. Use a modern RAID controller, such as the ServeRAID-6M SCSI controller or DS4800 Fibre Channel controller. Use RAID-10. Configure the largest stripe size available for your RAID controller. The VMFS3 uses a block size of 1 to 8 MB, so you should match that if possible. If using Fibre Channel, use two high-speed host adapters (such as the IBM Total Storage DS4000 FC2-133 HBA). Configuring your SAN might be a rather complex task (especially if you have a large fabric) but try to zone the ESX Server system(s) in such a way that they get a fast and direct access to their storage partitions. Generally speaking, sizing the correct disk subsystem is a time-consuming and complicated task and you should invest some time to analyze the planned usage and expected system load to properly size your storage configuration. Tip: If you are interested in deploying an ESX Server solution with IBM DS4000 series (FAStT) storage, see Implementing VMware ESX Server 2.1 with IBM TotalStorage FAStT, SG24-6434. For the network configurations, we suggest that you try at least two Gigabit Ethernet controllers available for use by the virtual machines (do not share these with the Console OS). However, as was the case with the disks, in certain circumstances such as some BladeCenter servers, the system might not have two Gigabit Ethernet adapters available and only one Gigabit controller will need to be used. The configuration of your network attachment largely depends on your network layout. It does not make sense to configure multiple Gigabit Ethernet adapters if the ESX Server server is attached to a 10/100 Mbps switch. Depending on your network layout, you might consider attaching the ESX Server system to a high-speed switch that has Link Aggregation Control Protocol (LACP) capabilities so that you can bond multiple network interface cards together (often referred to as trunking or channelling). The CPU subsystem should be sized according to the expected load of all virtual machines plus an overhead of 10% to 20% (the virtualization overhead can vary
Chapter 13. VMware ESX Server
429
depending on workload, but go with 20% to be safe). Because ESX Server scales well up to 16-socket, you can easily go with a 16-way System x3950. Keep in mind, however, that performance tuning is not the only variable and that you must also take factors such as fault tolerance and maintenance into account. Tip: Hyper-Threading is supported with ESX Server. If you are using ESX Server 2.1.2 or later, we recommend that you enable it if you have less cores then the maximum supported. In order for Hyper-Threading to be enabled on an ESX Server system, it must be enabled both in the system BIOS as well as on the EXS Server configuration. If you are using ESX Server 2.1, however, we strongly recommend that you disable Hyper-Threading. Hyper-Threading has to be disabled in the server’s BIOS and the ESX Server kernel.
Firmware and BIOS settings We recommend that you use the latest UpdateXpress CD to update your BIOS and firmware to the latest levels. You can download UpdateXpress the latest from: http://www.ibm.com/systems/support/supportsite.wss/docdisplay?brandind= 5000016&lndocid=MIGR-53046
13.2.2 Manual NUMA tuning ESX Server features very good support for NUMA systems, such as the System x3950. If you followed our recommendations in 13.3.2, “The /proc file system” on page 447, your system might already be configured optimally. The intelligent, adaptive NUMA scheduling and memory placement policies in ESX Server can manage all VMs transparently, so that administrators do not need to deal with the complexity of balancing VMs between nodes by hand. However, manual override controls are also available and administrators with advanced skills can still optimize their systems as they see fit. These optimizations work seamlessly regardless of the type of guest operating systems that are running. ESX Server provides transparent NUMA support even to guests that do not support NUMA hardware, such as Windows NT 4.0. This unique feature of ESX Server allows clients to take advantage of cutting-edge hardware, even when they are tied to earlier operating systems.
Home nodes ESX Server assigns each VM a home node when the VM begins running. A VM will only run on processors within its home node. Newly-allocated memory
430
Tuning IBM System x Servers for Performance
comes from the home node as well. Thus, if a VM’s home node does not change, the VM uses only local memory and avoids the performance penalties that are associated with remote memory accesses to other NUMA nodes. New VMs are assigned to home nodes round-robin—the first VM goes to the first node, the second VM to the second node, and so on. This policy ensures that memory is used evenly throughout all nodes of the system. Several operating systems, such as Windows Server 2003, provide this level of NUMA support, which is known as initial placement. This support might be sufficient for systems that only run a single workload, such as a benchmarking configuration, which does not change over the course of the system’s uptime. However, initial placement is not sophisticated enough to guarantee good performance and fairness for a datacenter-class system that is expected to support changing workloads with an uptime measured in months or years. To understand the weaknesses of an initial-placement-only system, consider the following example: An administrator starts four VMs. The system places two of them on the first node and two on the second node. Now, consider what happens if both VMs on the second node are stopped or if they simply become idle. The system is then completely imbalanced with the entire load placed on the first node. Even if the system allows one of the remaining VMs to run remotely on the second node, it suffers a serious performance penalty because all of its memory remains on its original node.
Dynamic load balancing and page migration To overcome the weaknesses of initial-placement-only systems, ESX Server combines the traditional initial placement approach with a dynamic rebalancing algorithm. Periodically (every two seconds by default), the system examines the loads of the various nodes and determines whether it should rebalance the load by moving a virtual machine from one node to another. This calculation takes into account the relative priority of each virtual machine to guarantee that performance is not compromised for the sake of fairness. The rebalancer selects an appropriate VM and changes its home node to the least-loaded node. When possible, the rebalancer attempts to move a VM that already has some memory located on the destination node. From that point forward, the VM allocates memory on its new home node, unless it is moved again. It only runs on processors within the new home node. Rebalancing is an effective solution to maintain fairness and ensure that all nodes are fully utilized. However, the rebalancer might need to move a VM to a node on which it has allocated little or no memory. In this case, the VM will incur a performance penalty associated with a large number of remote memory accesses. ESX Server can eliminate this penalty by transparently migrating memory from the virtual machine’s original node to its new home node. The
Chapter 13. VMware ESX Server
431
system selects a page, 4 KB of contiguous memory, on the original node and copies its data to a page in the destination node. The system uses the VM monitor layer and the processor’s memory management hardware to seamlessly remap the VM’s view of memory, so that it uses the page on the destination node for all further references, eliminating the penalty of remote memory access. When a VM moves to a new node, ESX Server immediately begins to migrate its memory in this fashion. It manages the migration rate adaptively to avoid overtaxing the system, particularly when the VM has very little remote memory remaining or when the destination node has little free memory available. The memory migration algorithm also ensures that it will not move memory needlessly if a VM is moved to a new node for only a short period of time. When all these techniques of initial placement, dynamic rebalancing, and intelligent memory migration work in tandem, they ensure good memory performance on NUMA systems, even in the presence of changing workloads. When a major workload change occurs, for instance when new VMs are started, the system takes time to readjust, migrating VMs and memory to new, optimal locations. After a short period of time, the system completes its readjustments and reaches a steady state.
Manual NUMA controls Some administrators with advanced skills might prefer to control the memory placement and processor utilization by hand. This can be useful, for example, if a VM runs a memory-intensive workload, such as an in-memory database or a scientific computing application with a large data set. Such an application can have performance improvements if 100% of its memory is allocated locally, while VMs managed by the automatic NUMA optimizations often have a small percentage (5% to 15%) of their memory located remotely. An administrator might also want to optimize NUMA placements manually if the system workload is known to be simple and unchanging. For example, an eight-processor system running eight VMs with similar workloads would be easy to optimize by hand. ESX Server provides two sets of controls for NUMA placement, so that administrators can control both memory and processor placement of a VM. The ESX Server Web-based Management User Interface (MUI) which is only available in ESX Servers prior to 3.x, allows you to indicate that a VM should only use the processors on a given node through the Only Use Processors option and that it should only allocate memory on the desired node through the Memory Affinity option. If you set both of these before a VM starts, it only runs on the desired node, and all of its memory is allocated locally. An administrator can also move a VM manually to another node after the VM has started running. In this case, the Page Migration Rate of the VM should also be set manually, so that memory from the VM’s previous node can be moved to its new node. The ESX Server documentation includes a full description of how to set these options.
432
Tuning IBM System x Servers for Performance
Note that manual NUMA placement can interfere with the ESX Server resource management algorithms, which attempt to give each VM a fair share of the system’s processor resources. For example, if 10 VMs with processor-intensive workloads are placed manually on one node and only two VMs are placed manually on another node, then it is impossible for the system to give all 12 VMs equal shares of the system’s resources. You should take these issues into account when using manual placement. For example, if your server workload requires specific CPU affinity (that is, associate the virtual machines to specific NUMA nodes, such as the x3950), you can use the VMware management interface to manually distribute the virtual machines evenly over the number of NUMA nodes of your server. See Figure 13-1 for the CPU affinity selection screen. Running 64 virtual machines on a 16-way System x3950 4-node complex would require you to assign 16 virtual machines per 4-way node. The memory balancing is optimized automatically according to the static CPU allocation that you made so that a virtual machine running on NUMA node 1 does not use the memory in NUMA node 4.
Figure 13-1 Setting the CPU affinity for individual virtual machines
Tip: Check the files in the /proc/vmware/sched/ directory to get information about the behavior of your NUMA system.
Chapter 13. VMware ESX Server
433
Real-life example of NUMA algorithms Figure 13-2 shows a simple real-life example of what happens when the VMkernel uses the algorithms that we have described. We created a 4-way virtual machine on a 2-node 8-way system with single core CPUs and Hyper-Threading enabled. We stressed the 4-way virtual machine to simulate a heavy workload to push it towards full utilization of the resources that were associated to that. Figure 13-2 shows the esxtop utility.
NUMA node 1
8 logical CPUs in node 2 (HT)
NUMA node 2
8 logical CPUs in node 1 (HT)
Figure 13-2 The esxtop utility running on a NUMA system
This basic example illustrates that the workload generated by the 4-way virtual machine is kept local in a single NUMA node (NUMA node 1 in this case).
13.2.3 VMware disk partitioning In ESX Server, you should differentiate between two types of disk storage: Disk storage for the virtual machines Disk storage for the ESX Server kernel, the SWAP file, the various log files, and the Console OS.
434
Tuning IBM System x Servers for Performance
You have to use VMFS for the partition on which your virtual machines reside. Generally, the default settings proposed by the ESX Server installer are well suited and should not need further optimization. Example 13-1 shows a typical disk layout with some external storage (SAN or an EXP400). Example 13-1 Sample disk layout
/dev/sda1 /dev/sda2 /dev/sda3 /dev/sda4 /dev/sda5 /dev/sda6 /dev/sdb1
*
1 7 326 517 517 4430 1
6 47974+ 325 2552319 516 1528191 4442 31411926 4429 31307881+ 4442 103981+ 17681 142022601
83 Linux /boot 83 Linux / 82 Linux swap f Extended partition fb VMFS fc VMFS for core dump and swap fb VMFS for virtual machines
Note that the size of the Console OS swap partition is set to at least twice the amount of maximum suggested Console OS memory. This setting allows you to add more virtual machines to your server. If you know the size of the Console OS memory requirements exactly, you can also set this partition to just twice the amount of memory you effectively are using. On your external storage, configure more than one VM file system if the external storage is very large. While you might lose some efficiency in storage usage, you will have some added resiliency in case one VM file system is corrupted. However, having multiple VM file system partitions on a relatively small disk array only decreases overall performance because the disk heads have to move back and forth every time another VM file system partition is accessed.
13.2.4 Tuning the VMware kernel The VMware kernel has several tuning options that can impact your overall system performance significantly. In this section, we explore some of the important tuning parameters of the ESX Server kernel.
Page sharing ESX Server features an algorithm to share equal memory pages across multiple virtual machines and thus reduce the total amount of used memory on the ESX Server system. Page sharing has little to no performance impact (about 2%) and might even speed up page lookups. The benefits of page sharing are largely workload dependent. We recommend you leave page sharing enabled, but if you think you might be able to increase server performance by disabling page sharing, edit the /etc/init.d/vmware file and add a -m before the -n as shown in Example 13-2.
Chapter 13. VMware ESX Server
435
Disabling page sharing, however, might mean an increase in memory demand (although less with Linux guests than with Windows guests). Example 13-2 Disabling page sharing in /etc/init.d/vmware
“cd”$vmdb_answer_SBINDIR”&& “$vmdb_answer_SBINDIR”/”$kernloader”-m -n”$maxCPU” “$vmdb_answer_LIBDIR”/”$kernel’||exit 1”
Setting network speeds It is generally better to change all network interface cards that are used in the ESX Server system from auto-negotiate to full duplex. The same applies to all involved hubs or switches because the auto-negotiate setting often results in less than optimal network speeds. There are situations where the Broadcom Console OS driver set to auto-negotiate in many cases does not negotiate properly when connected to a 100 Mbps network. The network is generally still functional, however, the performance is only 1/100th of the expected bandwidth. Setting the correct network speed is important. You can set the network speed of the network interface cards that are associated to the Console OS in the /etc/modules.conf file as shown in Example 13-3. Note: The commented lines in this example refer to network cards that are assigned to the virtual machines. Example 13-3 Setting the network adapter speed in /etc/modules.conf
alias parport_lowlevel parport_pc alias scsi_hostadapter aic7xxx alias eth0 e100 e100_speed_duplex=4 alias scsi_hostadapter ips #alias eth1 eepro100 alias scsi_hostadapter1 aic7xxx alias scsi_hostadapter2 aic7xxx #alias usb-controller usb-ohci alias scsi_hostadapter ips alias scsi_hostadapter ips The correct settings for your particular network interface card are displayed in the examples section at the end of the /etc/modules.conf file or in the readme file of your network interface cards. You can also set the speed and duplex of the network cards that are associated to virtual machines in the VMware management interface. Log in to the VMware
436
Tuning IBM System x Servers for Performance
management interface as root and go to the Network Connections menu in the options section. Figure 13-3 shows an example of setting the speed of the network interfaces.
Figure 13-3 Setting the network speed in the VMware management interface
Network link aggregation ESX Server 2.x supports network link aggregation following the IEEE 802.3ad standard. You can, therefore, aggregate the bandwidth of multiple network interface cards to one single larger link that supports not only link aggregation but also fault tolerance. To configure this feature, you need multiple network interface cards of the same type and a network switch that supports Link Aggregation Control Protocol (LACP).
Storage configuration You can make a the following changes to optimize your storage I/O: Fibre channel queue depth on QLogic HBAs VMware recommends in high I/O environments that you increase the HBAs maximum queue depth. You can make this change in the Service Console by
Chapter 13. VMware ESX Server
437
editing the hwconfig file. Step-by-step instructions are provided in the KB Answer 1267. You adjust the queue depth in the /etc/vmware/hwconfig file. Search for a line that has the following statement in it (the numbers and the name of the QLogic HBA might vary on your system): device.x.x.x.name = “QLogic Corp QLA2300 64-bit FC-AL Adapter (rev 01)" Example 13-4 shows an example of setting the queue depth to 64. Example 13-4 Set the HBA queue length in /etc/vmware/hwconfig
device.7.3.0.class = "0c0400" device.7.3.0.devID = "2300" device.7.3.0.name = "QLogic Corp QLA2300 64-bit FC-AL Adapter (rev 01)" #Add the queue depth statement here device.esx.7.3.0.options = "ql2xmaxqdepth=64" #For previous versions of VMware ESX the statement is below device.vmnix.7.3.0.options = "ql2xmaxqdepth=64" device.7.3.0.subsys_devID = "0009" device.7.3.0.subsys_vendor = "1077" device.7.3.0.vendor = "1077" The default value is 16 but the recommended value to use is 64. While it is possible to change this setting for Emulex based HBAs, VMware has no specific recommendation to due so. Therefore, we would recommend only changing this setting with QLogic HBAs. Outstanding disk requests per virtual machine This setting goes along with the queue depth setting. VMware recommends that you change both of them at the same time. You make this change by changing the Disk.SchedNumReqOutstanding value in the Options → Advanced Settings in the MUI. Change this setting to match what was changed in the HBA maximum queue depth change (64 is the recommended value). The VMware KB Answer 1268 has step-by-step instructions for changing this setting. http://www.vmware.com/support/kb/enduser/std_adp.php?p_faqid=1268
VMware kernel swapping The swap mechanism of the VMware kernel allows you to run significantly more virtual machines on a single server. However, you will see an increase in disk I/O when the system starts to swap data out to the disk.
438
Tuning IBM System x Servers for Performance
For optimal performance, you should monitor the swap file of the VMware kernel closely, and reduce the number of active virtual machines or install more memory as soon as the VMware kernel starts to swap data out to disk. The VMware swap file should ideally not be used by the system during normal operation. To minimize the impact if it is used, we suggest that you create a VMFS partition on your local disks and put the swap file there. Tip: Keep an eye on the data in the /proc/vmware/swap/stats file and make sure that the used swap value does not constantly exceed zero (0).
Note: If you install ESX Server on a blade server that uses IDE drives, you need to put the VMware kernel swap file to an external storage because VM file system volumes are not supported on IDE drives.
13.2.5 Tuning the virtual machines In contrast to the very good default settings of the VMware kernel, tuning the virtual machines can offer performance improvements. Depending on the workload that you run within your virtual machines, the tuning tips we discuss in this section might make a noticeable improvement the virtual machine’s performance. It should also be noted here that any tuning of the operating system that you run in the virtual machine will quite likely also have a positive effect on the overall system performance. Tip: We suggest you always install the VMware tools and the drivers that come with them. Doing so increases the performance and decreases the overall load on the ESX Server system.
Tuning the VM memory allocation When you create a new virtual machine, you are required to select the amount of allocated memory. Similar to the physical memory that is installed in a dedicated server, a operating system running in a virtual machine cannot use memory that was previously allocated. Therefore, if you allocate a certain amount of memory to your virtual machine but the operating system and application running within demand more memory, swapping will occur. As always, swapping caused by a lack of memory is mostly undesired because fast memory access times are replaced with relatively slow disk access times. Thus, we suggest that you size the memory available to the virtual machine according to the total needs of both the operating system and the application that is installed in the virtual machine.
Chapter 13. VMware ESX Server
439
ESX Server offers you some advantages in sizing the memory allocation: You can always allocate more memory to a virtual machine than it will effectively use. Unused memory is shared among all other virtual machines. You can resize the allocated memory of a virtual machine easily. However, always keep in mind that a change in memory of a running virtual machine requires a reboot of that virtual machine.
Selecting the right SCSI driver You can create virtual machines with two different types of (virtual) disk controllers: The BusLogic driver is the default driver for Windows NT4 and Windows 2000 and the LSI driver is the default for Windows 2003 which features compatibility with a broader range of operating systems. The BusLogic driver that is shipped with all supported guest operating systems performs adequately with small files of up to 1 KB in size. If you have an application that makes use of such small files, the default BusLogic driver might be the best choice for you. Using the BusLogic driver that is supplied by the guest operating systems does not, however, yield the best possible performance. VMware strongly recommends upgrading the operating system provided driver with the version that is available for download from the VMware support site. The LSI Logic controller driver shows a significant performance improvement for larger files. However, this driver might not be available for all supported guest operating systems. VMware provides disk images with the drivers for Linux and Windows operating systems so that you can install the virtual machine with the diskette drive.
Disabling any unused devices ESX Server offers a comprehensive set of virtualized hardware. However, most of the time, your system will not use all devices. For example, consider an Apache server. As soon as you set up this system, you will have no further need for the CD-ROM, the serial ports, or the parallel interface. Windows operating systems tend to poll these devices from time to time. This behavior can cause
440
Tuning IBM System x Servers for Performance
processor peaks on your ESX Server system because the polling causes an interrupt to occur. Generally, you should disable the following devices if your application or guest operating system do not make use of them:
COM Port 1 COM Port 2 LPT Port 1 CD-ROM (disable auto-detect in the VM) Tip: To disable the CD-ROM auto-detect feature of Windows Server 2003, edit the registry value HCCU\Software\Microsoft\Windows\CurrentVersion\ Policies\Explorer and set NoDriveTypeAutoRun to 0x000000FF.
Network interface driver ESX Server provides every virtual machine with a default network interface card that shows up in the virtual machine as an AMD PCNet adapter. This network interface card is supported by all ESX Server supported operating systems. While the default network interface card driver provides compatible and reliable connection to the network, the performance delivered by this network interface card is comparably low. After you install VMware Tools, you have the option of using the VMware Network Interface driver instead. This network interface provides remarkably better performance. Tip: Should you encounter problems with your network connection, switch back to the more compatible AMD network interface card.
Tuning for terminal servers Virtual machines that run a high number of concurrent processes, such as terminal servers can gain additional performance, by specifying their workload in the virtual machine configuration.
Chapter 13. VMware ESX Server
441
To specify the workload of the virtual machine, go into the virtual machine management interface and set the server type, as shown on Figure 13-4. Although this server type is called terminal server, this setting might also speed up other server types that run several concurrent processes. Changing this setting requires that you restart the VM before the change takes effect.
Figure 13-4 Set the workload for Terminal Services
Tip: In ESX Server Version 1.5.2 up to 3.0, you can perform the same optimization, but you have to open the virtual machine configuration file and add the following parameter: workload=terminalservices
Time synchronization It is important that ESX Server keep accurate time. To synchronize ESX Server with an NTP server, follow the directions as outlined in VMware KB Answer ID# 1339: http://www.vmware.com/support/kb/enduser/std_adp.php?p_faqid=1339 VMware also recommends that you synchronize your virtual machine’s time with the ESX Server’s time. This synchronization is a function of the VMware Tools that are installed in the virtual machines. For more detailed information about
442
Tuning IBM System x Servers for Performance
timekeeping, see the VMware white paper, Timekeeping in VMware Virtual Machines, which is available from: http://www.vmware.com/pdf/vmware_timekeeping.pdf
13.3 ESX Server 2.5 and later features The features that we describe in this section are specific to ESX Server 2.0 and later, but they also might be applicable to ESX Server 3. Note: Additional features that apply to ESX Server 2.0 and later are also included in 13.2, “General ESX Server tuning considerations” on page 427.
13.3.1 ESX Server 2.5 and later configuration best practices This section provides some guidance on the best ways to configure your system when running ESX Server 2.5 or later.
PCI card placement In NUMA configurations, performance can be impacted by the installation of PCI cards such as network adapters, Fibre Channel HBAs, and so on. To distribute the load equally, we recommend that you spread the placement of the adapters across all the nodes or chassis. Spreading the adapters also helps in the event of a node failure. Recommendations for PCI card placement include: Fibre Channel adapters On a 2-node system, for example, we recommended that you place one Fibre Channel HBA in node 1 and the other in node 2. We recommend a minimum of two HBAs to provide for redundancy in any ESX Server implementation. For QLogic based HBAs, it is recommended that you also change the Port Down Retry value in the QLogic BIOS to 15. Network controllers On a 2-node system, we recommend a minimum of four Gigabit Ethernet controllers: – One for the service console – One for VMotion – Two for a virtual switch for the virtual machines to use
Chapter 13. VMware ESX Server
443
On a 4-node system, we recommend a minimum of six network cards. On the x3950, because you have eight onboard Gigabit Ethernet controllers, we recommend you connect and use all of them. – One for the service console – One for VMotion – Four to six for virtual machines Table 13-2 shows how you would configure on the onboard NICs in a 2-socket or 4-socket x3950 with ESX Server 2.5 or later. Remember, in ESX Server 2.5 or later you are limited to 32 ports per virtual switch. So, with a 4-node configuration, we would have to create a minimum of two virtual switches. Also, you are limited to 8 Gigabit NICs in ESX Server 2.5 or later. So, if you want to use add-in PCI NICs, you must disable the onboard NICs in the BIOS. Note: These examples illustrate basic network configuration and do not take into account more advanced topics such as backup networks and DMZ networks. Table 13-2 Network configuration in ESX Server 2.5 or later
444
NIC
Node
Purpose
Label
vSwitch
NIC1
1
Service Console
eth0
none
NIC2
1
VMs
vmnic0 Outbound Adapter 0
vSwitch1
NIC3
2
VMotion
vmnic1 Outbound Adapter 1
VMotion
NIC4
2
VMs
vmnic2 Outbound Adapter 2
vSwitch2
NIC5
3
VMs
vmnic3 Outbound Adapter 3
vSwitch1
NIC6
3
VMs
vmnic4 Outbound Adapter 4
vSwitch2
NIC7
4
VMs
vmnic5 Outbound Adapter 5
vSwitch1
NIC8
4
VMs
vmnic6 Outbound Adapter 6
vSwitch2
Tuning IBM System x Servers for Performance
Hard drives We recommend that you install ESX Server on a RAID 1 array and add a hot spare drive for increased fault tolerance. The size of the hard drives that you need depends on how much RAM that you have in the server and how many virtual machines that you plan to run. We recommend that you configure your server with three 73 GB hard drives—two for a RAID 1 array and one hot spare. This configuration provides enough disk space for a 4-node x3950 with 64 GB of RAM and 64 virtual machines running. This configuration also assumes that all the virtual machines are running on the SAN and not local disk.
Disk partitioning Disk partition size depends on a number of factors including the number of virtual machines that will be running and the amount of RAM installed. Your swap should be two times the amount of RAM, and the VM file system 2 volume used for the VMkernel swap should be at least as large as the amount of physical RAM that is installed in the server. Table 13-3 shows an example of how to partition the disks for a 2-node, 8-way x3950 with 32 GB of RAM that is designed to run 32 virtual machines, assuming a 72.3 GB local RAID-1 array and with virtual machines stored on a SAN. Table 13-3 ESX Server 2.5 or later disk partitioning
Partition
Size
Comment
/boot
50 MB
Service Console boot files. Should be created as a primary partition.
/
4 GB
Root partition. Numerous problems can develop if the root partition runs out of disk space. Should be created as a primary partition.
swap
1 GB
Swap file for service console, should be twice the amount of RAM assigned to the service console. Should be created as a primary partition.
/var
1 GB
Various ESX Server logs are stored in this partition. This size should be sufficient to not run out of space. This size is also used if you plan to use the VMware method of a scripted install.
/home
512 MB
Virtual machine configuration files are stored here. They are small and this will be enough space regardless of how many virtual machines you have running.
Chapter 13. VMware ESX Server
445
Partition
Size
Comment
/vmimages
10 GB
This partition is can be used to store ISO images of various OS and application CDs that can then be mounted by the virtual machines.
VMFS2
32 GB
This partition will be formatted as the VMFS2 file type to create the VMkernel swap. This should be equal to the size of the physical RAM in the server. It can be made larger to allow more over allocation of memory.
core dump
100 MB
In the event of an ESX server crash, a log is put in the coredump partition to send to VMware support.
/tmp
1 GB
Optional. Some people like to create a partition for temp files
Service console memory and CPU For a 2-node x3950, set the service console RAM to 512 MB. For a 4-node x3950, set the service console RAM to 800 MB. If you plan on running additional applications within the service console, we recommend that you increase the RAM to 800 MB (this is the maximum) for all x3950 multi-node configurations. You should also take note of the minimum CPU value that is assigned to the service console. By default, when you install ESX Server 2.5 or later, it allocates 8% of CPU0 as the minimum for the service console. This allocation is based on the assumption that no additional applications were to be installed in the service console. Because we recommend that you install IBM Director Agent on large systems that are generally used with ESX Server, we also recommend that you increase the minimum CPU that is guaranteed to the service console. We recommend that you increase this amount to the following: 2-node configurations: 15% minimum CPU 4-node configurations: 30% minimum CPU Remember, these minimum values are only enforced if the service console needs the additional CPU cycles and if there is contention for resources. Under most circumstances the service console uses less CPU than the minimum listed here, and the unused processor capacity is available to virtual machines.
Network configuration We recommend that you start with all the network controllers set to auto negotiate for their speed and duplex settings. If you experience network-related performance issues you can try changing the NIC settings to 1000/Full. These
settings are in the MUI under Options → Network Connections. See Table 13-2 on page 444 for our recommended configuration using the onboard
446
Tuning IBM System x Servers for Performance
NICs, which is a basic configuration. For more information about advanced networking topics, VMware has several white papers about networking, available from: http://www.vmware.com/vmtn/resources/esx_resources.html
Additional best practices For additional best practices on ESX 2.5 or later, see the Best Practices for VMware ESX Server 2 white paper, which is available at: http://www.vmware.com/pdf/esx2_best_practices.pdf
13.3.2 The /proc file system In the Linux operating system, the /proc file system reveals important configuration information and also gives an administrator the ability to change several parameters while the system is running. The /proc file system also reveals some basic utilization figures. The /proc directory in the VMware Console OS, however, only includes information relevant to the Console OS not the entire ESX Server environment. The information that is related to the entire ESX Server environment can be found in the subdirectory /proc/vmware. Most performance data is stored in the respective statistics files in the various subdirectories, such as /proc/vmware/net/vmnic0. Of special interest are the following files: /proc/vmware/swap and /proc/vmware/sched, which displays the usage of the VMware kernel swap file and the NUMA behavior of the system (if applicable) /proc/vmware/sched/cpu, which identifies the affinity of every running virtual machine A more detailed discussion of the data available in /proc/vmware is in the ESX Server Administrator’s Guide, which is shipped with the product.
Chapter 13. VMware ESX Server
447
The sample output shown in Example 13-5 shows that the three virtual machines are allowed to run on any CPU (see the affinity column) but that the VMware scheduler has associated each virtual machine with a different processor to keep the load balanced (see the cpu column). Note: The file includes long lines. In Example 13-5, we have split the lines in two to make it easier to see the columns. Example 13-5 Output of the /proc/vmware/sched/cpu (split in two to show long lines)
vcpu vm type name
uptime
status
costatus
127 144 147 154
261486.327 261368.883 258486.650 189669.514
RUN WAITB WAITB WAITB
NONE NONE NONE NONE
wait NONE IDLE IDLE IDLE
127 144 147 154
SC V V V
console vmm0:Red_Hat vmm0:Red_Hat vmm0:Microso
waitsec cpu affinity 244159.128 251328.107 250898.313 175422.147
0 3 6 4
0 0,1,2,3,4,5,6,7 0,1,2,3,4,5,6,7 0,1,2,3,4,5,6,7
usedsec syssec 17306.070 3.698 10039.504 10.913 7599.642 19.104 14227.594 15.307
htsharing min max shares emin extrasec any any any any
8 0 0 0
100 100 100 100
2000 1000 1000 1000
100 100 100 100
0.000 2.855 0.000 7.561
Information in the /proc/vmware file system can often be cryptic. It also shows current values and offers no long-term analysis of the system performance.
13.3.3 Tuning the Console OS Because the Console OS is a very limited Red Hat Linux installation, there is not much room for improvement within the Console OS. Generally, no tuning of the Console OS should be necessary. The system already runs in run level 3 and has only the necessary services enabled.
448
Tuning IBM System x Servers for Performance
One tweak that might give a slight performance improvement is to disable some of the virtual terminals. To do this, you can comment out the lines that are associated with tty4, 5, and 6 in file /etc/inittab, as shown in Example 13-6. Example 13-6 The /etc/inittab file after commenting out terminals 4, 5 and 6
# Run gettys in standard runlevels 1:2345:respawn:/usr/sbin/vmkstatus tty1 2:2345:respawn:/sbin/mingetty tty2 3:2345:respawn:/sbin/mingetty tty3 #4:2345:respawn:/sbin/mingetty tty4 #5:2345:respawn:/sbin/mingetty tty5 #6:2345:respawn:/sbin/mingetty tty6 Also keep in mind that installing IBM Director Agent on your Console OS has a major impact on the overall performance of the Console OS because IBM Director Agent relies heavily on Java. If you plan to install IBM Director Agent, add at least another 50 MB of memory to your Console OS through the VMware management interface. Add more memory if you are adding other agents. While there are not many tuning parameters for the Console OS for a small ESX Server installation, there are some important settings to know if your ESX Server system hosts more than 60 virtual machines or if your ESX Server server is under very heavy load. In such cases, the Console OS, especially the Web server that is hosted by the Console OS, might become very slow and unresponsive. Should this be the case, increase the Console OS memory by going into the VMware management interface and configuring a Console OS memory size of 500 MB to 800 MB (always keep the overhead by IBM Director in mind). If the VMware management interface is still slow, change the process priority of the http daemon (the process that hosts the VMware management interface). To do so, log into the VMware Console OS (either locally or through SSH) and issue the following command to determine which process ID (PID) is associated with that httpd daemon: ps -axw If you have the IBM Director Agent installed on the Console OS, you might want to use the following command instead, because the Java processes will fill the screen: ps -axw | grep httpd
Chapter 13. VMware ESX Server
449
Example 13-7 shows the output of this command. As you can see, the httpd daemon has a process ID of 1431. As soon as you get the corresponding PID, you can then reprioritize the httpd using the following command: renice -10 -p 1431 This command adjusts the priority of the httpd to the increase in workload. (You might need to adjust the priority to yet a lower priority, such as -15, depending on the actual workload.) Example 13-7 Reprioritize the http daemon
[root@xseries01 root]# ps -axw PID TTY STAT TIME COMMAND 1 ? S 0:03 init 1431 ? S 0:00 /usr/lib/vmware-mui/apache/bin/httpd -DSSL -DSSL_ONLY -DSTANDARD_PORTS -DESX -d /usr/lib/vmware-mui/apach 1166 pts/0 R 0:00 ps -axw [root@xseries01 root]# ps -p 1431 PID TTY TIME CMD 1431 ? 00:00:00 httpd [root@xseries01 root]# renice -10 -p 1431 1431: old priority 0, new priority -10 In addition to adding CPU time to the httpd daemon, you can also adjust the memory that is reserved for the Web server. If you have more than 80 virtual machines, you will want to raise the default memory reservation from 24 MB to a higher value. You can configure the actual shared memory segment that is used by Apache in the /etc/vmware/config file by specifying the new value in bytes. Example 13-8 shows an increase from 24 MB to 28 MB (see the last two lines). Example 13-8 Configuring the shared memory size for Apache in /etc/vmware/config
vmware.fullpath = "/usr/bin/vmware" control.fullpath = "/usr/bin/vmware-control" wizard.fullpath = "/usr/bin/vmware-wizard" serverd.fullpath = "/usr/sbin/vmware-serverd" serverd.init.fullpath = "/usr/lib/vmware/serverd/init.pl" # The setting below increases the memory shares available for the httpd mui.vmdb.shmSize = "29360128" ~ While the renice command takes effect immediately, you have to restart the httpd daemon to make use of the bigger memory share. To do this, use the following command at a BASH prompt: killall -HUP httpd
450
Tuning IBM System x Servers for Performance
To ensure that you can login under heavy workload, we recommend that you increase the timeout value that the VMware authentication uses from the default of 30 seconds to a higher value. You can perform this change again through the /etc/vmware/config file as shown in Example 13-9 (the last two lines). Example 13-9 Configuring the VMware authentication timeout in /etc/vmware/config
vmware.fullpath = "/usr/bin/vmware" control.fullpath = "/usr/bin/vmware-control" wizard.fullpath = "/usr/bin/vmware-wizard" serverd.fullpath = "/usr/sbin/vmware-serverd" serverd.init.fullpath = "/usr/lib/vmware/serverd/init.pl" mui.vmdb.shmSize = "29360128" # The setting below increases the login timeout to 2 minutes vmauthd.connectionSetupTimeout = 120 ~ You might also have to increase the memory limit for the vmware-serverd daemon. Because this involves a restart of the vmware daemon, you should definitely stop all virtual machines before proceeding. This limit is changed in the /etc/vmware/config file by adding the setting specified in Example 13-10 to increase the soft memory limit to 64 MB and the hard to 96 MB (see the last four lines). Note that you specify the value in KB. Example 13-10 Setting the soft memory and hard memory limits in /etc/vmware/config
vmware.fullpath = "/usr/bin/vmware" control.fullpath = "/usr/bin/vmware-control" wizard.fullpath = "/usr/bin/vmware-wizard" serverd.fullpath = "/usr/sbin/vmware-serverd" serverd.init.fullpath = "/usr/lib/vmware/serverd/init.pl" mui.vmdb.shmSize = "29360128" vmauthd.connectionSetupTimeout = 120 # The line below will alter the soft memory limit vmserverd.limits.memory = “65536” # The line below will alter the hard memory limit vmserverd.limits.memhard = “98304” ~ After you have finished editing the configuration file, you have to restart the vmware-serverd by issuing one of the following commands: shutdown -r now killall -HUP vmware-serverd
Chapter 13. VMware ESX Server
451
Important: These commands reboot ESX Server. You should shutdown all virtual machines before rebooting the server or restarting the serverd daemon.
13.3.4 ESX Server 2.5.x design This redbook is not designed to replace the documentation already available from VMware and other sources. For detailed information about how to install and use ESX Server 2.5 or later, see the documentation that is provided by VMware at: http://www.vmware.com/support/pubs/esx_pubs.html In discussing architecture and design, we assume that the environment consists of a minimum of two ESX Server systems, shared SAN storage, VirtualCenter, and VMotion.
Overview of ESX Server 2.5 or later specifications ESX Server 2.5 or later has the following specifications: Physical ESX Server: – – – – – – – – – –
16 logical processors per system 80 virtual CPUs in all virtual machines per ESX Server system 64 GB of RAM per ESX Server system Up to 8 swap files, with a maximum file size of 64 GB per swap file 64 adapters of all types per system Up to 8 Gigabit Ethernet or 16 10/100 Ethernet ports per system Up to 32 virtual machines per virtual switch 16 host bus adapters per ESX Server system 128 logical unit numbers (LUNs) per storage array 128 LUNs per ESX Server system
ESX Server 2.5 or later virtual machines: – – – – –
Up to two virtual CPUs per virtual machine with the optional vSMP module Up to 3.6 GB of RAM per virtual machine Up to four virtual SCSI adapters and up to 15 SCSI disks Virtual disk sizes up to nine TB Up to four virtual Ethernet network adapters
For the latest list of supported guest operating systems and qualified hardware see the Systems Compatibility Guide, which is available at: http://www.vmware.com/vmtn/resources/esx_resources.html
452
Tuning IBM System x Servers for Performance
Virtual Infrastructure with ESX Server 2.5 or later With ESX Server 2.5 or later and VirtualCenter 1.3 virtual infrastructure consists of the following components:
ESX Server 2.5 or later VirtualCenter 1.3 vSMP VMotion
ESX Server runs on a physical server, while VirtualCenter can either run on a separate physical server or in a virtual machine. One thing to consider if you choose to run VirtualCenter in a VM is that if the parent ESX Server system goes offline, you will not have access to VirtualCenter until the server is back online or until you restart the virtual machine on another host. vSMP and VMotion are features already installed and are unlocked with a license key. VMware offers a Virtual Infrastructure Node (VIN) license that includes the following software licenses:
ESX Server license Virtual SMP license VirtualCenter Agent license vMotion license
The VIN license offers considerable savings over buying all the individual licenses separately.
Number of servers and server sizing The number of servers that suit your needs is dependant on a several factors, including:
Scope of current project Future growth estimates High availability and disaster recovery plans Budgetary constraints
There are a number of different methods to try to calculate the number of ESX Serer systems you will need. Here are two of the more popular methods. The easiest rule of thumb is 4 to 5 virtual CPUs per physical CPU. This would result in 16 to 20 virtual machines per 4-way host or 32 to 40 per 8-way host, assuming that all were one vCPU virtual machines and low to moderate workloads. For memory, if you assume 1 GB per virtual machine. That should provide enough memory in most cases for virtual machines, the service console, and
Chapter 13. VMware ESX Server
453
virtualization overhead. If you plan on running multiple, memory-intensive workloads, consider increasing this number. From these calculations we can arrive at an 8-way (two-node) x3950 with 32 GB of RAM which could support 32 virtual machines, and a 16-way (4-node) x3950 with 64 GB of RAM, which could support 64 virtual machines. These calculations assume single-core CPUs. Because a dual-core CPU will not provide 100% the performance of 2 single-core CPUs, we recommend that you count a dual-core CPU as 1.5 physical CPUs, resulting in 6 to 7 virtual machines per CPU socket. If you are consolidating a number of existing physical servers, then another method that can be employed is to record the average peak CPU utilization of the physical machines and convert this into a total of MHz used. For example, if you have a physical server with two 500 MHz CPUs that have an average peak utilization of 50%, then your total would be 500 MHz of CPU for this system. To get an average peak utilization, you must record CPU utilization for at least a week during the normal hours when the application is being used. A month is recommended for the most accurate information. If you already use an enterprise monitoring tool such as IBM Tivoli, HP OpenView, NetIQ, and so on, then you might already have all the data you need. The next step is to add up the total CPU clock speeds of your ESX Server system. For example: A 2-node 8-way x3950 with 3 GHz CPUs would have a total of 24 000 MHz. a. From this total subtract 10% for the Console OS. This gives us 21 600 MHz. b. Subtract a certain amount for additional peak utilization and overhead; 20% is a safe number to use. This gives us 17 280 MHz to work with for our virtual machines. c. Divide that number against the 500 MHz of average peak utilization that we first determined. The yield is about 34 virtual machines (17 280/500 = 34.5). You can do similar calculations to determine how much memory you need as well. Take the average peak memory utilization of your physical servers, add 54 MB per system for virtualization overhead, and add 32 MB for any systems whose average peak is over 512 MB. This total is the amount of RAM that you need for your VMs. Then, add the amount of RAM that is assigned to the Service Console (512 MB would be an appropriate starting number on an 8-way ESX Server system), add 24 MB for the VMkernel, and this is the total amount of RAM needed.
454
Tuning IBM System x Servers for Performance
For example, if you had 10 physical systems to virtualize and each had an average peak memory utilization of 512 MB, then that would equal 5120 MB. Add 54 MB each for virtualization overhead (5120+540 = 5660 MB). This amount is the total amount of RAM for the VMs. Add 512 MB for the Service Console (5660+512 = 6172 MB) and 24 MB for the VMkernel (6172+24 = 6196) and this amount is the total amount of RAM that is needed to run these 10 VMs: 6 GB of RAM. Both methods provide very similar results in the number of virtual machines you could support on an 8-way x3950 server. Our experience is that in most organizations, these two methods usually result in a similar number of virtual machines per host. Therefore, to save yourself some time, we recommend you use the first method for initial sizing of your ESX servers. The exact mix of virtual machines and applications running will affect how many virtual machines you can run. Unfortunately there is no one formula that will calculate exactly how many virtual machines you can run. The low end of the recommendations that we illustrated here should provide a realistic and conservative target for most organizations. In reality, you could end up supporting more or fewer virtual machines. Future growth is harder to determine. The cycle that happens in many organizations which implement VMware’s Virtual Infrastructure model is: 1. At first, they are resistant to the idea of virtual machines. 2. After they see all the benefits of the virtual infrastructure and that they are not losing performance, the number of requests for new virtual machines can grow rapidly. 3. The result can be an over commitment of processor, memory, and I/O resources and a subsequent loss in overall performance. To avoid this cycle, one recommendation is that you follow the same purchasing, approval, and change management procedures for virtual machines as you do for physical systems. While the process can usually be streamlined and shorted for virtual machines, having a formal process in place to request virtual machines as well as a way to associate costs to each new virtual machine, you will have much better control over your virtual machine growth and a better idea of future growth. For more information about best practices, see VMware ESX Server: Scale Up or Scale Out?, REDP-3953, which is available from: http://www.redbooks.ibm.com/abstracts/redp3953.html
Chapter 13. VMware ESX Server
455
VMotion considerations When designing your virtual infrastructure, an important consideration is VMotion. VMotion is the feature that allows the migration of a virtual machine from one physical ESX Server system to another while the virtual machine is running. Because VMotion transfers the running architecture state of a virtual machine between two physical hosts, the CPUs of both physical hosts must be able to execute the same instructions. At a bare minimum, this means for VMotion to work your servers CPUs must be: Same vendor class (Intel or AMD) Same processor family (Pentium III, Pentium 4, Opteron, and so forth) Sometimes there are significant changes to processors in the same family that have different extended features, such as 64-bit extensions and SSE3. In these cases VMotion might not work, even though the CPUs are in the same processor family. CPU speed and cache level are not an issue, but the extended features will cause problems or VMotion failure if they are different on the target and host servers. For example, because the x366 and x260 use the same processors as the x3950, these servers would be suitable candidates for joining some x3950s in a VMotion configuration. However other System x servers with different processors will not. Another important requirement for VMotion is shared storage. The ESX Server systems across which you are going to run VMotion need to be zoned so that all LUNs are visible to all hosts.
Planning your server farm With VirtualCenter 1.0 and later, a farm is a group of ESX Server systems that can be used to organize your virtual infrastructure. A farm is also a VMotion boundary, meaning that all servers in a VMotion configuration must be defined in one farm. In your planning, you need to think about how many hosts are going to be in each farm. VMware recommends the following guidelines: No more than 16 ESX Server systems should be connected to a single VMFS volume. No more than 32 I/O-intensive virtual machines per LUN and no more than 100 low-I/O virtual machines per LUN. No more than 255 files per VMFS volume. Up to 2 TB limit on storage.
456
Tuning IBM System x Servers for Performance
Because VMotion requires shared storage, then the upper limit per farm would be 16 ESX Server systems per farm. You might want to create smaller farms for a number of reasons. The lower limit is two servers, assuming that you are using VMotion.
Storage sizing Similar to server sizing, there is no one universal answer that can be applied to every organization. There should not be more than 32 I/O-intensive virtual machines per VM file system volume, and staying within this limit should reduce any resource contention or SCSI locking issues. There are a number of ways to determine the most appropriate size of your VM file system volume. Here is one of the easier ways. Say that you have decided that two 8-way x3950 servers with 32 virtual machines on each server will meet your processing requirements. Using the 32 virtual machines per LUN guideline, you would need two LUNs for this configuration. If you create new virtual machines, you can estimate the average size of the virtual disks. If we use 20 GB of disk per VM, this would give us 640 GB per LUN. Consider adding a little additional space for growth—10% is a good rule of thumb—which brings us to 720 GB. If you are planning on using redo logs, you might want to add additional space for that as well.
Planning for networking There are various options when it comes to designing the networking portion of your server farm. The options chosen are often based on the characteristics of your physical network and networking and security policies of your company. One important factor is if a Gigabit Ethernet network is available. While not absolutely required for ESX Server, a Gigabit network is highly recommended. In ESX Server 2.5 or later, there are three basic components you should consider: Service console It is recommended that the service console have its own dedicated NIC for performance and security reasons. If you have a separate management network in your data center, then this is where you want to locate the service console NIC. In a default configuration, a 100 Mbps Ethernet controller is sufficient bandwidth for the service console. If you are planning on also using the service console for backups or other high bandwidth functions, then a Gigabit NIC is recommended.
Chapter 13. VMware ESX Server
457
Virtual machines The virtual machines use a separate network from the service console. A Gigabit network is not required but is highly recommended, because 32 virtual machines can generate significant network traffic. A good rule of thumb is 10 to 20 virtual machines per Gigabit Ethernet controller. This means that we need a minimum of 2 Gigabit Ethernet NICs for an 8-way x3950 running 32 VMs. Remember that this is the minimum recommendation. Adding one or two more virtual machines should guarantee enough network bandwidth available for all virtual machines. Another important consideration is if you have multiple VLANs in your data center that you want to make available to the virtual machines. When using multiple VLANs with ESX Server 2.5 or later, you have two options: – Install a physically separate NIC for every network that you want available. If you only have a few networks that you want to use, then this is a viable option. However, if you have 10 different networks, then this is obviously not a practical solution. Remember that ESX Server 2.5 or later only supports a maximum of eight gigabit network cards. – Use ESX Server’s support for VLAN tagging (802.1q). Using this option means that you can create a virtual switch with a separate port group for each VLAN that you want to use. If your physical switches support this, then this is the recommended option. One other consideration is redundancy for your virtual machine’s networking. With ESX Server 2.5 or later, you can have multiple NICs that are connected to one virtual switch not only to combine bandwidth but also to provide redundancy in case of a failure of one of the NICs or a physical cable. VMotion VMware lists a separate Gigabit Ethernet network as a requirement for VMotion. It is possible to use VMotion with a 100 Mbps network. However, performance might not be acceptable and it is not recommended. You should have a separate physical gigabit NIC for your VMotion network and a separate subnet created for VMotion to use. If you only have two systems running ESX Server, then it is possible to use a crossover cable between the two servers for a VMotion network. This is also useful for troubleshooting VMotion problems.
Network load balancing ESX Server 2.5 or later provides two methods for network load balancing for the virtual machines: MAC Out is the default method. Using this method requires no additional configuration in ESX Server. Simply connect two physical NICs to the virtual
458
Tuning IBM System x Servers for Performance
switch. No additional configuration on the physical switches is necessary. The only disadvantage of this method is that is it not very efficient. Often, most virtual machines end up using the same physical NIC and there is no method to select manually what physical NIC each virtual machine uses. IP Out is an optional way to configure your networking for better load balancing. The disadvantage to this method is that there is additional configuration steps required in ESX Server as well as your physical switches. You must configure your physical switches for 802.3ad (or EtherChannel in the case of Cisco switches). This is the recommended method for highest performance. This is a brief overview of networking with ESX Server 2.5 or later. Advanced topics such as backup networks, DMZ networks, traffic shaping, and detailed configuration steps are beyond the scope of this redbook. For in depth information about networking and configuration steps, see the documentation that is available on the VMware Web site: http://www.vmware.com/support/pubs/esx_pubs.html http://www.vmware.com/vmtn/resources/esx_resources.html
13.4 ESX Server 3.0 features The features that we describe in this section are specific to ESX Server 3.0. However, note that additional tuning features that are applicable to this section are also included in the 13.2, “General ESX Server tuning considerations” on page 427. VMware ESX Server 3.0 represents the next generation of virtual infrastructure products from VMware. With the release of these products, VMware offers its customers the opportunity to move beyond the consolidation of low utilization workloads to virtualizing their entire x86 based infrastructure. Larger workloads such as databases, messaging systems, CRM, ERP, and so forth are no longer considered bad candidates for virtualization starting with ESX Server 3.0. Features such as 4-way Virtual SMP, 16 GB of RAM available to virtual machines, as well as 32 logical CPUs with up to 128 virtual CPUs per ESX Server system will increase greatly the workloads that can be virtualized. The new features in ESX Server 3 and VirtualCenter 2 include the following: NAS and iSCSI support The ability to store virtual machines on lower cost NAS and iSCSI storage should allow more companies to take full advantage of all the features that VMware’s Virtual Infrastructure provide. Features such as VMotion and HA are supported on NAS and iSCSI.
Chapter 13. VMware ESX Server
459
Note: At the time of this writing, the specific NAS and iSCSI hardware to be supported by VMware was not available. 4-Way Virtual SMP Virtual SMP is an add-on that allows you to create virtual machines with more than one virtual CPU. With ESX Server 3.0, you have the ability to configure up to four virtual CPUs per VM with the new 4-way Virtual SMP option. This option allows the virtualization of larger work load applications such as database and messaging servers. 16 GB RAM for virtual machines ESX Server 3 allows you allocate up to 16 GB of memory per virtual machine. This combined with 4-way Virtual SMP allows for the virtualization of work loads and systems previously not allowed and provides the benefits of the Virtual Infrastructure environment to these systems. VMware High Availability An optional component of VirtualCenter 2, VMware HA (formerly known as Distributed Availability Services) detects failed virtual machines and automatically restarts them on alternate ESX Server hosts. With the virtual machine’s disks and configuration files residing on shared storage, failover time can be quite short. HA has added intelligence and rules that can be applied to restart VMs appropriately (for example, not restarting two load balanced virtual machines on the same ESX Server host). HA provides higher availability with the added cost or complexity of alternatives such as clustering. Distributed Resource Scheduler Another optional component to VirtualCenter 2, Distributed Resource Scheduler (DRS), allows for automatic balancing of virtual machines across your ESX Server hosts. DRS uses VMotion to migrate virtual machines from one ESX Server host to another when it detects that not enough resources are available for a given virtual machine. DRS still provides the ability to move virtual machines manually, as well as to override, or decline suggested VMotion activities. You will have the ability to exclude certain virtual machines from DRS, so that they can only ever be moved manually. DRS should allow for higher ESX Server utilization because workloads can be automatically migrated for optimal performance. VMware Consolidated Backup Consolidated backup is another optional component for ESX Server 3 and provides host-free, LAN-free, agentless backup of Windows virtual machines.
460
Tuning IBM System x Servers for Performance
This provides an easy way to backup an entire running virtual machine while allowing file-level restores. Consolidated backup works by acquiescing a virtual disk automatically before creating an online snapshot with no virtual machine downtime required. A separate physical machine can mount the snapshots and use a standard backup agent to back up the data. This also means it might be possible to remove backup agents from Windows virtual machines. Simplified Service Console ESX Server 3 has a new service console based on Red Hat Enterprise Linux 3. The new service console acts more like a standard virtual machine (with virtual I/O devices) consumes less resources and provides greater flexibility for installing third party applications with the service console. All storage and networking devices are now dedicated to the VMkernel meaning no need to divide devices between the service console and virtual machines. Service console resource needs are no longer dependant on the number of virtual machines running. VMFS3 With ESX Server 3 includes an updated file system that has a number of improvements including: – Improved disk locking to increase scaling for access by a larger number of ESX hosts to shared storage – Greater reliability due to distributed journaling – The flexibility to resize and add LUNs on the fly VMFS3 is no longer a flat file system, so you can create directories and subdirectories. Hot-add Virtual Disks ESX Server 3 provides the ability to add a virtual disk to a virtual machine while running. Multiple snapshots ESX Server 3 adds a multiple snapshot feature similar to what is available in the VMware Workstation 5 or later product. Large-scale management VirtualCenter 2 can manage hundreds of ESX Server hosts and thousands of virtual machines. VirtualCenter 2 is designed from the ground up to handle the largest virtual infrastructure deployments. Unified user interface ESX Server 3 and VirtualCenter 2 share a new VMware Virtual Infrastructure Client accessible from any Windows PC or browser. Remotely access and
Chapter 13. VMware ESX Server
461
manage ESX Server hosts, virtual machines and VirtualCenter Management Servers from the new VC client. ESX Server 3 no longer includes the MUI for management of the ESX server. Instead you connect to the ESX host from the new VirtualCenter client, or better, do all your administration directly from within VirtualCenter. Improved Virtual Infrastructure management VirtualCenter 2 centralizes storage of virtual machine configuration files and VMware licenses for greater deployment flexibility and ease of management. There is a new Licensing Server that can be installed on the VirtualCenter server, within a virtual machine or on a physical server, to manage all the licenses. Licenses will be allocated out of a pool. All virtual machine configuration files (.vmx, nvram, and so forth) are now stored on a shared VMFS volume instead of each individual ESX server. There is a new Web-based remote console to allow system administrators to connect to virtual machines through a Web browser instead of needing a remote console application installed. VirtualCenter 2 has enhanced performance graphs and counters. Improved security Access controls within VirtualCenter have been greatly expanded. Custom roles and permissions are now available. You will have much more flexibility in deciding who can control and change resources. VirtualCenter auditing has been improved to provide an accurate audit trail of who made what changes. Expanded ESX Server hardware support A broader range of hardware will be added to the hardware compatibility list (HCL) for ESX Server 3 allowing customers greater flexibility in their choice of hardware. New servers including more dual-core CPU models, more SAN products, NAS, iSCSI, as well as a wider range of I/O cards will be supported. NIC support increased to 20 Gigabit Ethernet adapters and 26 10/100 Ethernet adapters. Maximum logical CPUs supported has doubled from 16 to 32, helping to enable scale-up implementations such as a multi-node x3950 configuration. Improved Networking In addition to support for more physical NICs, virtual networking has bee improved by increasing the number of ports per virtual switch (vSwitch) to 1024 with up to 32 uplinks per vSwitch and a maximum of 1024 vSwitches per ESX Server system. Per-port NIC teaming, traffic shaping policies and new security policies greatly increase the flexibility of network configurations within ESX Server. Expanded ESX Server guest operating system support New operating systems will be supported as guests in ESX Server 3. Most notably Red Hat Enterprise Linux 4, with others expected as well.
462
Tuning IBM System x Servers for Performance
Improved support for Citrix Citrix was one of the high visibility applications that VMware targeted for improved performance. We were not able to do any Citrix testing for this redbook, but we have been told ESX Server 3 improves Citrix performance by up to 50%.
13.4.1 ESX 3.0 best practices Best practices are usually developed over a period of time by the user community and vendor working together. These best practices are modified continually as additional information related to ESX 3.0 tuning is used.
PCI card placement Similar to the ESX2.0 or later, in NUMA configurations with multiple nodes or chassis, performance can be affected by the installation of PCI cards such as network adapters, Fibre Channel HBAs, and so on. To distribute the load equally, we recommend that you spread the placement of the adapters across all the nodes or chassis. Spreading the adapters also helps in the event of a node failure. The following are best practices for PCI card placement on ESX 3.0: Fibre Channel adapters On a 2-node x3950, for example, we recommended that you place one Fibre Channel HBA in node 1 and the other in node 2. We recommend a minimum of two HBAs to provide for redundancy in any ESX Server implementation. For QLogic based HBAs, it is recommended that you also change the Port Down Retry value in the QLogic BIOS to 15. Network card configuration ESX Server 3 can support up to 1024 ports for each virtual switch (ESX Server 2.5.x was limited to 32 ports per virtual switch). It is no longer necessary to separate the console OS, virtual machines, and VMotion NICs. You can simply assign all physical NICs to the switch created during install and create new port groups for each. Figure 13-6 on page 466 shows the port configuration utility. We expect that VMware and its user community will develop a more comprehensive list of best practices for the extensive new networking features in ESX Server 3.
Chapter 13. VMware ESX Server
463
Service console memory and CPU resources With ESX Server 3, the amount of RAM that is assigned to the service console is no longer tied to the number of virtual machines. The default amount is 256 MB, as shown in Figure 13-5. You can change this setting after, but not during, the installation.
Figure 13-5 Service console memory
As with ESX Server 2.5 or later, you might want to increase the service console memory if you are running additional applications in the service console. We recommend that you increase it to 512 MB if you are running IBM Director Agent, backup clients, or other applications in the service console. Regarding CPU resources, although the service console in ESX Server 3 is designed to consume less resources, at this time we feel the minimum CPU settings recommended for ESX 2.5 or later can still be applied to ESX Server 3. We recommend that you increase this amount to the following: 2-node configurations: 15% minimum CPU 4-node configurations: 30% minimum CPU 8-node configurations: 50% minimum CPU
Disk drives and partitioning Because ESX Server 3 no longer requires a local VM file system partition for the VMkernel swap file, it is now possible to use smaller hard drives of the same size, no matter how many virtual machines that you are running or how much RAM is installed in the physical server.
464
Tuning IBM System x Servers for Performance
With ESX Server 3, disk partitioning is no longer dictated by how much RAM is installed in the physical server and how many virtual machines you have running. Table 13-4 shows the recommended partitioning scheme for ESX Server 3 on a multi-node System x3950. Table 13-4 ESX Server 3 partitioning
Partition
Size (MB)
Comment
/boot
100
Boot files. Should be primary partition.
/
4096
Root partition. Should be primary partition.
swap
1024
service console swap. Should be primary partition.
/var/log
1024
VMware log files
coredump
100
VMkernel coredump partition
/tmp
1024
Optional temp partition
VMFS3
Rest of disk
Could be used for virtual machines.
It is still recommended that the swap partition be twice the amount of RAM that is allocated to the service console. The other partitions will not need to change size based upon the number of virtual machines, amount of RAM, and so forth. As you can see, even on a multi-node x3950 with 32 GB or more of RAM, we could fit everything on a 18 GB disk as opposed to ESX Server 2.5 or later where we needed 72.3 GB disks. As shown in Table 13-4, we no longer need the /home or /vmimages partitions from ESX Server 2.5 or later because all the virtual machine configuration files as well as all your ISO files are now stored on shared storage, SAN, iSCSI, or NAS.
Chapter 13. VMware ESX Server
465
Network configuration Although it will take time for VMware and the user community to develop best practices regarding the new network features for ESX Server 3, we can initially recommend the following best practices for network configuration: Add your additional NICs to the first virtual switch that was created during the installation and add ports to this switch for your virtual machines, VMotion, NAS storage, and so forth (Figure 13-6).
Figure 13-6 ESX Server 3 virtual switch port groups
466
Tuning IBM System x Servers for Performance
Change the default security policy on the vSwitch to Reject for all three: Promiscuous, MAC Change, and Forged Source (Figure 13-7). Unless you have a need for any of these in your organization, it is probably best for security reasons to set to disallow.
Figure 13-7 vSwitch security settings
Storage configuration One recommendation that we can make at this time is that if you plan on using iSCSI storage, you should use a hardware-based initiator supported by VMware if available and not the software-based initiator that is built into ESX Server. The iSCSI HBA has a TCP Offload Engine that offers higher performance and much less overhead than using the software-based initiator.
Service console firewall Another new feature in ESX Server 3 is that the service console firewall is enabled by default. In its default setting, all ports that are needed for communication to and from VirtualCenter are open. If you are not going to run any additional applications or agents within the service console, you can leave the firewall as is and get the added security without any configuration. However if you plan to install additional applications (for example IBM Director Agent), you must reconfigure the firewall. For more information, see the Virtualization on the IBM System x3950 Server, SG24-7190: http://www.redbooks.ibm.com/abstracts/sg247190.html
Chapter 13. VMware ESX Server
467
For initial testing purposes or troubleshooting you might want to disable the firewall temporarily. You can do this by logging onto the service console and entering the following command: esxcfg-firewall -u This command disables the firewall until the next reboot. Example 13-11 shows the options that you can use with the esxcfg-firewall command. Example 13-11 Output of the esxcfg-firewall -help command esxcfg-firewall -q|--query -q|--query <service> -q|--query incoming|outgoing -s|--services -l|--load -r|--resetDefaults -e|--enableService <service> -d|--disableService <service> -o|--openPort <port,tcp|udp,in|out,name> -c|--closePort <port,tcp|udp,in|out> --blockIncoming --blockOutgoing --allowIncoming --allowOutgoing -h|--help
468
Tuning IBM System x Servers for Performance
Lists current settings. Lists setting for the specified service. Lists setting for non-required incoming/outgoing ports. Lists known services. Loads current settings. Resets all options to defaults Allows specified service through the firewall. Blocks specified service Opens a port. Closes a port previously opened via --openPort. Block all non-required incoming ports (default value). Block all non-required outgoing ports (default value). Allow all incoming ports. Allow all outgoing ports. Show this message.
Part 4
Part
4
Monitoring tools In this part, we introduce the performance monitoring tools that are available to users of System x servers. We describe the tools that are specific to each of the three operating systems as well as Capacity Manager, a component of IBM Director. We also provide detailed instructions that show how to use these tools. This part includes the following chapters:
Chapter 14, “Windows tools” on page 471 Chapter 15, “Linux tools” on page 537 Chapter 16, “ESX Server tools” on page 579 Chapter 17, “Capacity Manager” on page 591
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
469
470
Tuning IBM System x Servers for Performance
14
Chapter 14.
Windows tools This chapter introduces some of the tools that are available to an administrator of a Windows Server 20031 server for performance tuning. It also covers Capacity Manager, which is part of IBM Director and which is discussed in detail in Chapter 17, “Capacity Manager” on page 591. The tools that we discuss in this chapter include:
14.1, “Performance console” on page 472 14.2, “Task Manager” on page 505 14.3, “Network Monitor” on page 511 14.4, “Other Windows tools” on page 520 14.5, “Windows Management Instrumentation” on page 522 14.6, “VTune” on page 528
How you use these tools depends on your performance tuning needs. Each tool provides its own unique capabilities, and each tool has its advantages and disadvantages. In Chapter 19, “Analyzing bottlenecks for servers running Windows” on page 655, we use the information in this chapter to explain how to detect bottlenecks.
1
Product screen captures and content reprinted with permission from Microsoft Corporation.
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
471
14.1 Performance console The Performance console is a valuable monitoring tool that Windows administrators commonly use to monitor server performance and to isolate bottlenecks. The tool provides real-time information about server subsystem performance, and the data collection interval can be adjusted based on your requirements. The logging feature of the Performance console makes it possible to store, append, chart, export, and analyze data captured over time. Products such as SQL Server and Exchange provide additional monitors that allow the Performance console to extend its usefulness beyond the operating system level.
14.1.1 Comparing Performance console with Capacity Manager For the purpose of monitoring the performance of your System x server running Windows Server 2003, the Performance console and Capacity Manager are available. Sometimes, it is more appropriate to use one tool over the other, based on your monitoring objectives. You would use the Performance console in the following circumstances: You can run the Performance console during activity-intensive periods to get a real picture of server performance. For example, you might want to measure your Windows Active Directory domain controller performance in the early morning when user logins storm the server. Alternatively, Capacity Manager gathers data all the time, including during idle periods that might not apply to this particular need. You can use the Performance console to monitor networking-related counters effectively, such as NetBEUI, NWLink, TCP/IP, network utilization, and others. This type of monitoring is important if you are monitoring how these protocols effect your network subsystem performance. Capacity Manager captures only general network counters such as packet rate, errors, and others. The Performance console can send alerts when predefined threshold levels are reached. This type of monitoring is useful especially when you want to perform actions as soon as your pre-set threshold conditions are met. With Capacity Manager, you can only view exceeded thresholds after you gather data from the test systems and generate a report. With the Performance console, you can view special counters that are provided by Microsoft BackOffice® applications. For example, when you install Exchange Server, it installs additional object counters of its own. You can then monitor and analyze these counters and relate them to your Exchange Server’s performance.
472
Tuning IBM System x Servers for Performance
You would use Capacity Manager in the following situations: Capacity Manager is always collecting data, while with the Performance console, you have to start the logging process first. A month’s worth of data is always available to the administrator from all systems that are running IBM Director (including non-Windows systems). So, performance and capacity analysis is easy and convenient. Also, in the Performance console, collecting data into a log file over a long period of time could produce a very large file that could use much of your disk space. Capacity Manager is a tool that lets you easily compare many servers. The Performance console is for analyzing a single server or a few servers in real time. The advantage of Capacity Manager is that it collects many Performance console statistics and others that are not available elsewhere into a database that can be evaluated offline. For example, to manage 50 servers, Capacity Manager allows the administrator to collect data automatically on all 50 servers using a scheduled request. Then, the results can be dumped into a database, and the administrator can study all 50 servers to see which deserves attention. By default, Capacity Manager saves one month of data. You can merge multiple months of data together, which allows you to evaluate a longer period of time or to compare the results month-to-month. This type of data comparison is not easily done with the Performance console. From where should you monitor performance? To better manage the system overhead produced by the Performance console, we recommend that you run it in the following way: If you are not monitoring network performance, monitor your servers remotely from another system, such as a Windows XP workstation. If you are monitoring network performance, do so locally on the server, so as not to add extra network utilization by capturing performance data from a remote workstation.
Chapter 14. Windows tools
473
14.1.2 Overview of the Performance console window The Performance console includes two tools: System Monitor Performance Logs and Alerts Figure 14-1 shows the main Performance console window.
Figure 14-1 The Windows Server 2003 Performance console
The Performance console is a snap-in for Microsoft Management Console (MMC). You can use the Performance console to access the System Monitor and Performance Logs and Alerts tools. You open the Performance console by clicking Start → Administrative Tools → Performance or by typing PERFMON on the command line.
474
Tuning IBM System x Servers for Performance
Tip: If there is no Administrative Tools folder, you can display it as follows: 1. Right-click Start and click Properties. 2. At the Taskbar and Start Menu properties dialog box, click Customize. 3. Select the Advanced tab. 4. Scroll down the Start menu items list until you find the System Administrative Tools section. 5. Select the Display on the All Programs menu and the Start menu option. 6. Click OK to close the Customize Start Menu dialog box. 7. Click OK to close the Taskbar and Start Menu properties dialog box. When starting the Performance console on a Windows Server 2003, the System Monitor runs automatically. The default monitors are: Memory: Pages/Sec PhysicalDisk: Avg. Disk Queue Length Processor: % Processor Time Windows 2000 Server does not start System Monitor automatically. You need to do this manually. You can use the System Monitor to view real-time or logged data of objects and counters. You can use Performance Logs and Alerts to log object and counters and to create alerts. Displaying the real-time data of objects and counters is sometimes not enough to identify server performance. Logged data can provide a better understanding of the server performance. You can configure alerts to notify the user or to write the condition to the system event log based on thresholds.
Chapter 14. Windows tools
475
System Monitor Figure 14-1 on page 474 shows the Windows Server 2003 System Monitor interface. There are three ways to view the real-time or logged data counters: Chart This view displays performance counters in response to real-time changes or processes logged data to build a performance graph. Histogram This view displays bar graphics for performance counters in response to real-time changes or logged performance data. It is useful for displaying peak values of the counters. Report This view displays only numeric values of objects or counters. You can use it to display real-time activity or logged data results. It is useful for displaying many counters. To edit the view, right-click in the main window of the System Monitor and select Properties. On the General tab, you can change the view. We discuss System Monitor in detail in 14.1.3, “Using System Monitor” on page 481.
476
Tuning IBM System x Servers for Performance
Performance Logs and Alerts The Performance Logs and Alerts window (Figure 14-2) lets you collect performance data manually or automatically from local or remote systems. You can display saved data in System Monitor or export data to a spreadsheet or database.
Figure 14-2 The Performance console: Performance Logs and Alerts
Performance Logs and Alerts provides the following functions: Counter logs This function lets you create a log file with specific objects and counters and their instances. You can save log files in different formats (file name + file number or file name + file creation date) for use in System Monitor or for exporting to database or spreadsheet applications. You can schedule the logging of data, or you can start the counter log manually using program shortcuts. You can also save counter log settings in HTML format for use in a browser either locally or remotely through TCP/IP. Trace logs You can use trace logs to debug applications or system events such as DNS, the Local Security Policy, and the Active Directory. Trace logs supply more detailed information than you can obtain in counter logs. Unlike counter logs, you cannot view the log file itself with System Monitor. You need to use a separate tool such as TRACERPT.EXE. This utility was introduced with Windows Server 2003 (see Table 14-6 on page 520). You can use TRACERPT.EXE to generate both a detailed comma-delimited text file and a summary file. Example 14-1 show the summary report that is obtained using TRACERPT.EXE on an existing trace log.
Chapter 14. Windows tools
477
Example 14-1 Trace log summary obtained using TRACERPT.EXE Files Processed: Test Trace Log_000003.etl Total Buffers Processed 67 Total Events Processed 3797 Total Events Lost 0 Start Time 12 July 2004 End Time 12 July 2004 Elapsed Time 50 sec +-------------------------------------------------------------------------------------+ |Event Count Event Name Event Type Guid | +-------------------------------------------------------------------------------------+ | 265 DiskIo Read {3d6fa8d4-fe05-11d0-9dda-00c04fd7ba7c}| | 913 DiskIo Write {3d6fa8d4-fe05-11d0-9dda-00c04fd7ba7c}| | 1 SystemConfig PhyDisk {01853a65-418f-4f36-aefc-dc0f1d2fd235}| | 40 SystemConfig Services {01853a65-418f-4f36-aefc-dc0f1d2fd235}| | 1 SystemConfig Video {01853a65-418f-4f36-aefc-dc0f1d2fd235}| | 1 SystemConfig Power {01853a65-418f-4f36-aefc-dc0f1d2fd235}| | 1 SystemConfig LogDisk {01853a65-418f-4f36-aefc-dc0f1d2fd235}| | 2 SystemConfig NIC {01853a65-418f-4f36-aefc-dc0f1d2fd235}| | 1 SystemConfig CPU {01853a65-418f-4f36-aefc-dc0f1d2fd235}| | 2 EventTrace Header {68fdd900-4a3e-11d1-84f4-0000f80464e3}| | 129 Thread Start {3d6fa8d1-fe05-11d0-9dda-00c04fd7ba7c}| | 388 Thread DCStart {3d6fa8d1-fe05-11d0-9dda-00c04fd7ba7c}| | 123 Thread End {3d6fa8d1-fe05-11d0-9dda-00c04fd7ba7c}| | 393 Thread DCEnd {3d6fa8d1-fe05-11d0-9dda-00c04fd7ba7c}| | 1 Process Start {3d6fa8d0-fe05-11d0-9dda-00c04fd7ba7c}| | 1 Process End {3d6fa8d0-fe05-11d0-9dda-00c04fd7ba7c}| | 34 Process DCStart {3d6fa8d0-fe05-11d0-9dda-00c04fd7ba7c}| | 34 Process DCEnd {3d6fa8d0-fe05-11d0-9dda-00c04fd7ba7c}| | 539 TcpIp Send {9a280ac0-c8e0-11d1-84e2-00c04fb998a2}| | 587 TcpIp Recv {9a280ac0-c8e0-11d1-84e2-00c04fb998a2}| | 3 TcpIp Connect {9a280ac0-c8e0-11d1-84e2-00c04fb998a2}| | 10 TcpIp Disconnect {9a280ac0-c8e0-11d1-84e2-00c04fb998a2}| | 24 TcpIp Retransmit {9a280ac0-c8e0-11d1-84e2-00c04fb998a2}| | 3 TcpIp Accept {9a280ac0-c8e0-11d1-84e2-00c04fb998a2}| | 175 UdpIp Send {bf3a50c5-a9c9-4988-a005-2df0b7c80f80}| | 126 UdpIp Recv {bf3a50c5-a9c9-4988-a005-2df0b7c80f80}| +-------------------------------------------------------------------------------------+
478
Tuning IBM System x Servers for Performance
You can log operating system or application activity using either system or non-system event providers. For system providers, the following events are provided by the Windows Server 2003 kernel trace provider: – – – – – –
Process creations/deletions Thread creations/deletions Disk input/output Network TCP/IP Page faults File details
Non-system providers can be application events. You can also code provider events. Any application or service that is installed in your system can support event providers. For example, Active Directory, NetLogon, and Local Security Authority are non-system event providers. Alerts This function lets you track objects and counters to ensure that they are within a specified range. If the counter’s value is under or over the specified value, an alert is issued. Actions from an alert include: – – – –
Sending the alert to another machine Logging the alert in the application event log Starting a new counter log Running a command from the command line
Alerts can be started and stopped automatically or manually. See 14.1.4, “Using performance logs and alerts” on page 486 for more information.
Objects, counters, and instances An object in System Monitor is any component that generates performance data. There are many objects built into Windows Server 2003. Each hardware component in your system is an object: processor, memory, hard drives, network cards, and other components in your machine. Objects are not only hardware components but also software components. Terminal services, Routing and Remote Access services, database server, and e-mail server applications that are installed on your system can have objects in System Monitor.
Chapter 14. Windows tools
479
Each object has one or more counters. For example, the processor object has, among others, the following counters:
%Processor Time %User Time Interrupts/sec %Interrupt Time Each counter can have multiple instances, which means there can be more than one of the same counter for that object. For example, in a multi-homed server, there will be multiple instances of network adapters, as illustrated in Figure 14-3.
Figure 14-3 Objects, counters and instances
In summary, the object is the hardware or software component in your machine, the counter is the value of a specified object that can be measured by System Monitor, and the instance identifies all or each of the members of the object.
480
Tuning IBM System x Servers for Performance
14.1.3 Using System Monitor To create a chart in System Monitor, you select the performance objects and configure the view. When you select a system object for display, the values of the specified counter are put into a chart in graphical format, as shown in Figure 14-4.
Figure 14-4 Multi instance object chart view
Table 14-1 explains the values that are included in this chart view. Table 14-1 Chart view values
Value
Description
Last
The latest value of the selected counter
Average
The average value of the selected counter
Minimum
The minimum value of the selected counter
Maximum
The maximum value of the selected counter
Duration
The period of time that you measure
Color
The selected color for the counter
Scale
The multiplier that you use to calculate the graphical value from the actual value
Counter
The performance values of the selected object
Instance
The member of the selected object
Parent
The upper level object of the selected object
Object
Hardware or software component
Computer
The name of the computer where you get the object
Chapter 14. Windows tools
481
Figure 14-5 shows the System Monitor toolbar.
Figure 14-5 System Monitor toolbar
Table 14-2 describes the options that are available from the System Monitor toolbar. Table 14-2 System monitor toolbar icons
Button
482
Function
Description
New counter set
Clears all the selected counters
Clear display
Clears all data samples
View current activity
Displays current activity
View log file data
Displays activity from log file
View chart
Displays performance data using line graphics
View histogram
Displays performance data using bar graphics
View report
Displays performance data using names and numbers
Add counter
Adds a new counter to System Monitor
Remove counter
Removes the selected counter from the counter list
Highlight
Highlights the selected counter
Copy properties
Copies all the graph properties and counter list to new windows
Paste counter list
Pastes all of the copied counter list to a new window
Properties
To change System Monitor properties
Freeze display
Pauses display in System Monitor
Update data
Updates data instantly
Help
Help for the active performance function
Tuning IBM System x Servers for Performance
The difference between the chart view and the histogram view (shown in Figure 14-6) is that the chart view shows the current and historical values in a line graph and the histogram view shows only the current values in a bar chart. The histogram view is useful for displaying or tracking values of the selected counters when you are only interested in the current value (that is, when you are not interested in what value the counter showed the last time it was sampled).
Figure 14-6 System Monitor, histogram view
The difference between the chart view and the report view is that the report view uses only text and numbers. If you are tracking many values and you need to reduce system overhead, you should use the report view. Tip: The report view gives a quick overview of the status of a server. This view is especially useful when you are performing an initial analysis of a poorly performing system and you want to determine quickly which counters are outside the range of good values. For more information, see Chapter 18, “Spotting a bottleneck” on page 627. You can change the information that the report displays using the General tab of the System Monitor properties. You can choose how each counter is displayed (for example, real time, average, minimum, or maximum).
Chapter 14. Windows tools
483
Figure 14-7 System Monitor, report view
Adding counters To create a chart that includes the objects that you want to monitor, follow these steps: 1. Click the Add counter icon ( ) on the toolbar or right-click the System Monitor and select Add Counters to open an Add Counters dialog box, as shown in Figure 14-8. From here, you can select the performance object, counters, and instances that you want to monitor.
Click Explain for information about the counter.
Figure 14-8 Add Counters window with the Explain window
2. Select the computer name that you want to monitor (local or remote).
484
Tuning IBM System x Servers for Performance
3. Select the Performance object that you want to add. 4. Select the specific counters that you want to monitor or click All counters to select all the counters for the selected object. 5. Select the specific instances that you want to monitor or click All instances. Selecting Total shows the total amount of activity that is generated by all instances. 6. Click Add. 7. Repeat steps 3 to 6 until you have selected all the performance objects, counters, and instances in which you are interested, and then click Close.
Deleting objects If you no longer want to monitor a counter, you can delete it as follows: 1. Select the counter from the counter list at the bottom of the main menu. You can only select the counter while in the chart or histogram view. 2. Click the Delete icon (
) in the toolbar or press Delete on your keyboard.
Note: If you want to clear all the counters from the counter list, click the New counter set icon ( ) in the toolbar. If you want to clear the System Monitor chart samples, click the Clear display icon ( ) in the toolbar.
Saving object and counter settings You can save objects, counters, and instances settings to an HTML file. You can then open the HTML file from any Windows Server 2003 system with Internet Explorer® and continue viewing the performance data in real time. You can also save the chart as a report (.TSV) that you can export to a spreadsheet or word processor. To save the report to file, right-click the window and click Save As. Specify the file type, location, and file name.
Highlighting an object counter If you are working with multiple objects and counters in a graph, sometimes it is hard to differentiate or focus on a particular counter, especially if you are using the chart view. To highlight a particular counter: 1. Select the counter from the counter legend list. 2. Click the Highlight icon (
) in the toolbar or press Ctrl+H.
Chapter 14. Windows tools
485
14.1.4 Using performance logs and alerts Logs are useful in capturing and storing data to disk for analysis at a later time. You can also collect data from multiple systems into a single log file. You can collect different counters, or counters of the same type, from multiple machines. You can use the data for comparative analysis for different machines with the same counters or analysis of only one machine with its counters. Creating a log consists of selecting objects, counters, and instances or selecting all counters and all instances of a specific object and starting and scheduling the log capture. You can load this data back into System Monitor for analysis. If you are using many counters and the sample frequency is too small, the log file requires a large disk space. If you are collecting multiple counters from a remote machine, this time this process might affect your network performance. You can collect data from remote machines in two ways: Collect all the data from remote machines using one workstation, which is the easiest way to collect the remote data but can effect network performance. Configure all the remote machines to store data on their own local disks and collect it through batch files or scripts. Alerts let you track objects and counters to ensure that they are within a specified range. If the counter’s value is under or over the specified value, an alert is issued. Actions from an alert include:
Sending the alert to another machine Logging the alert in the event log Starting a new counter log Running a command from the command line
Counter logs There are two types of logs: Counter logs You can create a log file with specific objects and counters and their instances. Log files can be saved in different formats (file name + file number or file name + file creation date) for use in System Monitor or for exporting to database or spreadsheet applications. You can schedule the logging of data, or you can start the counter log manually using program shortcuts. You can also save counter logs settings in HTML format for use in a browser, either locally or remotely through TCP/IP.
486
Tuning IBM System x Servers for Performance
Trace logs You can create trace logs that include trace data provider objects. Trace logs differ from counters logs in that they measure data continuously rather than at specific intervals. We discuss trace logs in “Trace logs” on page 497.
Toolbar Figure 14-9 illustrates the toolbar icons for working with logs. Delete
Export (Save as)
Properties
New log settings
Start the selected log
Stop the selected log
Figure 14-9 Performance Logs and Alerts: counter logs toolbar
Creating a new counter log To create a new counter log: 1. From the Performance console, select Counter Logs. 2. Click the New log settings icon ( Figure 14-10.
) from the toolbar, as shown in
Figure 14-10 Counter Logs
3. Enter a new log name for the counter log and then click OK.
Chapter 14. Windows tools
487
4. The New Counter Log dialog window opens, as shown in Figure 14-11.
Figure 14-11 New counter log, General tab
5. You now have the choice of adding objects or counters to the log. Click Add Objects to add an object and all its associated counters, or click Add Counters to add individual counters. Whichever option you choose, you can select the computer to monitor and then select the relevant objects or counters that you want to capture. 6. In the General tab, use Sample data every to set how frequently you want to capture the data. If you capture many counters from a local or remote computer, you should use long intervals. Otherwise, you might run out of disk space or consume too much network bandwidth. 7. In the Run As field, input the account with sufficient rights to collect the information about the server to be monitored, and then click Set Password to input the relevant password.
488
Tuning IBM System x Servers for Performance
8. On the Log Files tab (Figure 14-12), set the type of the saved file, the suffix that is appended to the file name, and an optional comment. You can use two types of suffix in a file name: numbers or dates. (Table 14-3 lists the log file types.) If you click Configure, then you can also set the location, file name, and file size for a log file.
Figure 14-12 New counter log, Log Files tab Table 14-3 Counter log file formats
Log file type
Description
Text file - CSV
Comma-delimited log file (CSV extension). Use this format to export to a spreadsheet.
Text file - TSV
Tab-delimited log file (TSV extension). Use this format to export the data to a spreadsheet program.
Binary file
Sequential, binary log file (BLG extension). Use this format to capture data intermittently (stopping and resuming log operation).
Binary circular file
Circular, binary-format log file (BLG extension). Use this format to store log data to same log file, overwriting old data.
SQL Database
Logs are output as an SQL database.
Chapter 14. Windows tools
489
9. On Schedule tab (Figure 14-13), specify when this log is started and stopped. You can select the option box in the start log and stop log section to manage this log manually using the Performance console shortcut menu. You can configure to start a new log file or to run a command when this log file closes.
Figure 14-13 New counter log, Schedule tab
Starting and stopping a counter log When creating a counter log, you can schedule the start and stop time, or you can specify whether to start or stop the counter log manually. To start and stop the counter log manually, do the following: 1. Select the counter log that you want to start. 2. Click the Start the selected log icon (
) on the toolbar.
3. To stop the counter log, click the Stop the selected log icon ( toolbar.
) on the
Tip: We recommend that you configure the Schedule tab to both start and stop any counter logs so that you can eliminate the problem of filling up a hard drive if you forget to stop the log manually.
490
Tuning IBM System x Servers for Performance
You can also use the menu to start and stop the logs, as shown in Figure 14-14.
Figure 14-14 Counter logs menu
Saving counter log settings You can save the counter log settings to use them later. To save log settings, do the following: 1. Select the counter log file in which you want to save settings. 2. Right-click the log. The window shown in Figure 14-14 opens. 3. Click Save Setting As. 4. Select a location and enter a file name, and then click Save (saving to an HTML file is the only option). You can then open this log settings file using Internet Explorer.
Deleting a counter log If you no longer want to use any counter log, you can delete it as follows: 1. Select the counter log that you want to delete. 2. Click the Delete icon (
) in the toolbar or press Delete on your keyboard.
Importing counter logs properties You can import counter logs settings from saved files. To import settings, do the following: 1. Right-click the right-hand window. 2. Select New Log Settings From. 3. The Open dialog window opens. Choose the location and select a file name, and then click Open. 4. The Name dialog box opens. If you want to change the log setting name, you can do so. Otherwise, click OK.
Chapter 14. Windows tools
491
5. A dialog window opens where you can add or remove counters and change log file settings and scheduling. 6. If you want to edit the settings, change the required fields. Otherwise, click OK.
Retrieving data from a counter log file After you have saved data to a log file, you can retrieve that data and process it. By default, System Monitor displays real-time data. In order to display previously logged data, do the following: 1. Click the View log data file icon ( Figure 14-5).
) on the System Monitor toolbar (shown in
2. The System Monitor Properties dialog box opens at the Source tab. Select Log Files and then click Add. The Select Log File dialog box opens. Select the log file that you want and click Open. 3. At the System Monitor Properties dialog box, select the Data tab. You should now see any counter that you specified when setting up the Counter Log. If you only selected counter objects then the Counters section is empty. To add counters from an object, simply click Add... and then select the appropriate ones. Selecting a time frame: Depending on how long the counter log file was running, there can be quite a lot of data to observe. If you are interested in looking at a certain time frame when the log file was recording data, complete these steps: 1. Click the Properties icon (
) on the System Monitor toolbar.
2. The System Monitor Properties box opens. Click the Source tab. 3. Select the time frame that you want to view (Figure 14-15) and click OK.
Figure 14-15 Selecting a time frame
492
Tuning IBM System x Servers for Performance
Alerts This function lets you track objects and counters to ensure that they are within a specified range. If the counter’s value is under or over the specified value, an alert is issued.
Toolbar Figure 14-16 shows the toolbar for alerts. Delete
New alert settings
Properties
Start the selected alert
Stop the selected alert
Export List
Figure 14-16 Alerts toolbar
Creating an alert To create an alert that includes the objects that you want to track, follow these steps: 1. From the Performance console, expand Performance Logs and Alerts and click Alerts.
Figure 14-17 The Performance console showing alerts
2. Click the New alert settings icon (
) in the toolbar.
3. Enter a name for the new alert.
Chapter 14. Windows tools
493
4. An alert setting dialog opens (Figure 14-18). In the General tab, enter a comment that describes the new alert.
Figure 14-18 New alert settings, General tab
5. Click Add and the Add counter window opens (similar to Figure 14-8) where you select the computer and the objects, counters, and instances that you want to monitor. 6. Specify the threshold values for each counter that you want to monitor. 7. Specify how often you want the Performance console to monitor the counter. 8. In the Run As field, enter the account with sufficient rights to collect the information about the server that you want to monitor and then click Set Password to input the relevant password.
494
Tuning IBM System x Servers for Performance
9. On the Action tab (Figure 14-19), enter information to log an entry in the application log, to send a message to a specified machine, to start a new performance data log, or to run a program.
Figure 14-19 New alert settings, Action tab
10.On the Schedule tab (Figure 14-20), set when the alert scan starts or stops. Within this tab, you can choose to start a new scan. Note that by default the alert starts as soon as you have finished setting the preferences. Unless you specify differently, the alert continues indefinitely. 11.When you are satisfied with your settings, click OK in the alert properties dialog box. If you did not change the schedule settings then the alert should start.
Chapter 14. Windows tools
495
Figure 14-20 New alert settings, Schedule tab
Saving alert settings You can save the alert settings to create a template for tracking your system. To save alert settings, do the following: 1. Right-click the alert and select Save Settings As from the menu. 2. Choose a location and enter a file name, then click Save. You cannot change the file type because only HTML files are supported.
Starting and stopping an alert You can configure an alert to start at a specified time and stop at a specified time. You can also start and stop alerts manually. To start an alert manually, follow these steps: 1. Select the alert setting that you want to start. 2. Click the Start the selected alert icon ( alert and select Start from the menu.
) in the toolbar or right-click the
3. To stop the alert, select the Stop the selected alert icon ( ) in the toolbar or right-click the alert and select Stop from the menu. When an alert is stopped, its actions are disabled as well.
496
Tuning IBM System x Servers for Performance
Importing alert settings You can import alert settings from saved files. To import settings, do the following: 1. Right-click the right-hand pane. 2. Select New Alert Settings From. 3. The Open dialog window opens. Select the appropriate file and click Open. 4. The Name dialog box opens. If you want to change the alert settings name, do so. Otherwise, click OK.
Deleting alert settings If you no longer want to use any alert settings, you can delete them as follows: 1. Select the alert setting you want to delete. 2. Click the Delete icon ( ) in the toolbar, press Delete on your keyboard, or right-click the setting and select Delete from the menu.
Trace logs Figure 14-21 shows the trace logs toolbar. Delete
New log settings
Properties
Start the selected log
Stop the selected log
Figure 14-21 Trace logs toolbar
Chapter 14. Windows tools
497
Creating a trace log To create a new trace log, perform the following steps: 1. From the Performance console, expand Performance Logs and Alerts and click Trace Logs.
Figure 14-22 The Performance console showing trace logs
2. Click the New log settings icon ( ) in the toolbar. 3. Enter a new log name for the trace log and then click OK. 4. The New Trace Log dialog window opens, as shown in Figure 14-23.
Figure 14-23 Trace Logs window, General tab
498
Tuning IBM System x Servers for Performance
5. In the General tab, you have to choose one of the providers. If you select Events logged by system provider, you can trace events logged by the operating system provider. There are some default counters, as shown in Figure 14-23. You can also select Page faults and File details, but these providers require significant system resources. If you choose a non-system provider, you must add at least one counter to the trace. Programs can be non-system providers. 6. In the Run As field, enter the account with sufficient rights to collect the information about the server to be monitored and then click Set Password to input the relevant password. 7. On the Log Files tab (Figure 14-12 on page 489) set the type of the saved file, the suffix that is appended to the file name, and an optional comment. You can use two types of suffix in a file name: numbers or dates. Table 14-4 on page 500 lists the log file types. If you click Configure, then you can also set the location, file name, and file size for a log file.
Figure 14-24 Trace logs window, Log Files tab
Chapter 14. Windows tools
499
Table 14-4 Trace logs log file formats
Log file type
Description
Circular trace file
(ETL extension) Used to store data in the same file. Overwrites new records on old data.
Sequential trace file
(ETL extension) Used to store multiple same-size log files. Data is written to this type of log file until it reaches a user-defined limit; data is then written to the new log file.
8. On the Schedule tab (Figure 14-25) specify when this log is started and stopped. You can select Manually in the start log and stop log section to manage this log manually using the Performance console shortcut menu. You can configure it to start a new log file or run a command when this log file closes.
Figure 14-25 Trace logs window, Schedule tab
500
Tuning IBM System x Servers for Performance
9. In the Advanced tab, you can set the buffer settings. Buffer settings are important when you are storing data in trace logs, because data is stored in memory buffers and then transferred to a trace log file.
Figure 14-26 Trace logs window, Advanced tab
You can set the following settings in the Advanced tab: Buffer size: size of the buffer (in KB) when you use trace data Minimum number of buffers: the smallest number of buffers when you use trace data Maximum number of buffers: the largest number of buffers when you use trace data To flush the buffers periodically, select Transfer data from buffers to log file every and specify the transfer interval. Tip: If you want to check the installed providers and their status, select Provider Status in the General tab.
Chapter 14. Windows tools
501
Key objects and counters The key objects in Windows Server 2003 are: Memory Processor Disk Network
These objects form the basic set of Windows Server 2003 performance objects. You might need to monitor other objects. Some counters are disabled by default because of system overhead. Refer to Chapter 19, “Analyzing bottlenecks for servers running Windows” on page 655.
Monitoring disk counters In Windows Server 2003, there are two kinds of disk counters: Physical counters monitor single disks and hardware RAID arrays. Logical drive counters monitor software RAID arrays. Both counters are enabled by default, which was not the case with Windows 2000. In Windows 2000, the physical disks were enabled and the logical disks were disabled by default. Use the DISKPERF command to enable or disable these counters. Enter DISKPERF -? for help with this command. Note: You should use physical drive counters if the system is using hardware RAID such as the IBM ServeRAID adapter.
Monitoring network counters Counters for monitoring network activity are activated by default on Windows Server 2003. This is not the case with Windows 2000. To monitor network-specific objects in Windows 2000, you need to install the Network Monitor Driver: 1. 2. 3. 4. 5. 6. 7. 8.
502
Open the Network and Dial-up Connections in the Control Panel. Select any connection. Click File → Properties. In the General tab, click Install. Select Protocol. Click Add. Select Network Monitor Driver. Click OK then Close.
Tuning IBM System x Servers for Performance
Using the Performance console with other tools You can use the following applications with a performance chart and logs to provide better reports or analysis data:
Internet Explorer Spreadsheet tools Word processing tools Database servers
You can export log file data to spreadsheets or databases to provide better data management. You can open saved settings file with Internet Explorer (which requires Windows 2000 or later). You can also add System Monitor Control to word processing applications. You can monitor data with this added control, as shown in Figure 14-27.
Design mode icon
More Control icon Figure 14-27 Implementing System Monitor in Microsoft Word
Microsoft Word You can send an active performance chart using Microsoft Word. Provided that the Microsoft Word user has permission (and authority) to access the server where the chart was created, real-time data can be displayed. This process requires at least Microsoft Word 97. To add this control: 1. Open a new Microsoft Word document and place the cursor where you want to insert this control. 2. On the toolbar, select View → Toolbars → Control Toolbox.
Chapter 14. Windows tools
503
3. Click the More Controls icon and then select System Monitor Control, as shown in Figure 14-27. Note: Control is added in design mode. In design mode, you can use this control as a Visual Basic® object. If you click the Design Mode icon in the control toolbox, you can use this control as System Monitor.
Internet Explorer You can easily monitor servers from a remote location using Internet Explorer. To monitor a remote location, do the following: 1. Prepare chart settings as described in 14.1.3, “Using System Monitor” on page 481. 2. When you are monitoring performance, right-click the chart and then select Save as, as described in “Saving object and counter settings” on page 485. 3. Copy this file to a remote location. 4. Open this file using Internet Explorer. 5. The snapshot window opens. Click the Freeze Display icon ( toolbar (Figure 14-28) to restart the chart view. The chart view should now be running in the browser window.
Figure 14-28 Running System Monitor in Internet Explorer
504
Tuning IBM System x Servers for Performance
) in the
Missing performance objects If you cannot find the object that you want to add, refer to “Key objects and counters” on page 502. If you still have a problem, check to be sure that disabled objects and counters do not appear in the Add Counters dialog box when you want to use them. If they do, follow the procedures in described in “Key objects and counters” on page 502. If you still have a problem with counters, follow these steps: 1. Open the Registry Editor. 2. Change the appropriate value under HKEY_LOCAL_MACHINE\SYSTEM\ CurrentControlSet\Services\Service_name\Performance\ DisablePerformanceCounters from 1 to 0. Important: You should back up the registry before making any changes. If you edit the registry incorrectly, you can cause severe damage to your system.
14.2 Task Manager In addition to the Performance console, Windows Server 2003 also includes Task Manager—a utility that allows you to view the status of processes and applications and gives you some real-time information about memory usage.
14.2.1 Starting Task Manager You can run Task Manager using any one of the following methods: Right-click a blank area of the task bar and select Task Manager. Press Ctrl+Alt+Del and click Task Manager. Click Start → Run and type taskmgr. Task Manager has five views:
Applications Processes Performance Networking Users
This discussion focuses on the Processes, Performance, and Networking tabs.
Chapter 14. Windows tools
505
14.2.2 Processes tab Figure 14-29 shows the Processes tab.
Figure 14-29 Task Manager: Processes
In this view, you can see the resources that are consumed by each of the processes currently running. You can click the column headings to change the sort order on which that column is based.
506
Tuning IBM System x Servers for Performance
Click View → Select Columns to display the window shown in Figure 14-30. From this window, you can select additional data to display for each process.
Figure 14-30 Task Manager: select columns for the Processes view
Table 14-5 shows the columns that are available in the Windows Server 2003 operating system. Table 14-5 Columns in the Windows Server 2003 Processes view
Column
Description
Image Name
The name of the process. It can be the name of the EXE, but this is not necessary.
PID
Process Identification Number, an internally assigned number.
CPU Usage
Current CPU utilization. When the system is not doing any work, the process “System Idle Process” will be near 100%.
CPU Time
Total amount of time this process has used since it was started, in seconds.
Memory Usage
The total amount of memory used by the process, in KB. It includes both the paged and nonpaged pool memory used.
Memory Usage Delta
The change in memory usage since the last Task Manager update.
Peak Memory Usage
The peak amount of memory, in KB, used by the process since it was started.
Chapter 14. Windows tools
507
508
Column
Description
Page Faults
The number of times data had to be retrieved from disk for this process because it was not found in memory. This is a total since the process was started.
USER Objects
The number of Window Manager (USER) objects currently used by the process.
I/O Reads
The number of read input/output (file, network, and device) operations generated by the process.
I/O Read Bytes
The number of bytes read in input/output (file, network, and device) operations generated by the process.
Session ID
The ID of the session running the process.
User Name
The name of the user running the process.
Page Faults Delta
The change in the number of page faults since the last update.
Virtual Memory Size
The size, in KB, of the process’s share of the paging file.
Paged Pool
The paged pool (user memory) usage of each process. The paged pool is virtual memory available to be paged to disk. It includes all of the user memory and a portion of the system memory.
Non-Paged Pool
The amount of memory reserved as system memory and not pageable for this process.
Base Priority
The process’s base priority level (low/normal/high). You can change the process’s base priority by right-clicking it and selecting Set Priority. This remains in effect until the process stops.
Handle Count
The number of handles used by the process.
Thread Count
The number of threads this process is running.
GDI Objects
The number of Graphics Device Interface (GDI) objects used by the process.
I/O Writes
The number of write input/output operations (file, network, and device) generated by the process.
I/O Write Bytes
The number of bytes written in input/output operations (file, network, and device) generated by the process.
I/O Other
The number of input/output operations generated by the process that are neither reads nor writes (for example, a control type of operation).
I/O Other Bytes
The number of bytes transferred in input/output operations generated by the process that are neither reads nor writes (for example, a control type of operation).
Tuning IBM System x Servers for Performance
By adding the relevant columns, it is very easy to determine if a particular application is behaving improperly. For example, it is possible to see if a particular application has a memory leak simply by adding the Virtual Memory Size column. After the column has been added, note the value in the Virtual Memory Size column for the relevant application. You can log off or lock the server console at this point. After a while, depending on how frequently your server is running out of RAM, you can recheck this value to see if it has grown. If this Virtual Memory Size always rises and never comes down then it is possible that the relevant application has a memory leak.
14.2.3 Performance tab The Performance tab shows you performance indicators, as shown in Figure 14-31.
Figure 14-31 Task Manager: Performance
The charts show you the CPU and memory usage of the system as a whole. The bar charts on the left show the instantaneous values, and the line graphs on the right show the history since Task Manager was started.
Chapter 14. Windows tools
509
The four sets of numbers under the charts are as follows: Totals: – Handles: current total handles of the system – Threads: current total threads of the system – Processes: current total processes of the system Physical Memory: – Total: total RAM installed (in KB) – Available: total RAM available to processes (in KB) – File Cache: total RAM released to the file cache on demand (in KB) Commit Charge: – Total: total amount of virtual memory in use by all processes (in KB) – Limit: total amount of virtual memory (in KB) that can be committed to all processes without adjusting the size of the paging file – Peak: maximum virtual memory that is used in the session (in KB) Kernel Memory: – Total: sum of paged and nonpaged kernel memory (in KB) – Paged: size of paged pool that is allocated to the operating system (in KB) – Nonpaged: size of nonpaged pool that is allocated to the operating system (in KB) Tip: If you have a hyper-threaded processor, it shows as two CPUs on the Performance tab of Windows Task Manager.
510
Tuning IBM System x Servers for Performance
Networking tab Microsoft has introduced a couple of new views to Windows Task Manager with Windows Server 2003. One of those views is that of the Networking tab (Figure 14-32). The Networking tab is available if you have network issues.
Figure 14-32 Task Manager: Networking
14.3 Network Monitor Network Monitor is a useful tool that is shipped with Windows Server 2003. It captures network traffic for display and analysis. This tool makes troubleshooting complex network problems easier and more economical. With Network Monitor, you can capture network traffic from a local NIC. You can also attach remote stations over the network or through dial-up that are running the agent software. Network Monitor works by placing the Ethernet controller in promiscuous mode so that it passes every frame on the wire to the tracing tool. The tool supports capture filters so that only specific frames are saved for analysis. Network Monitor is a useful troubleshooting tool, and it also helps administrators in their daily work. It shows what types of protocols are flowing across the
Chapter 14. Windows tools
511
network, delivers a better understanding of how bandwidth is used, and detects security risks (viruses, unapproved client applications, and so forth). The version of Network Monitor that ships with Windows Server 2003 allows the capture of traffic only on the local server. However, the version included with Windows Server 2003 System Management Server (SMS) allows you to monitor any machine on your network.
14.3.1 Installing Network Monitor To install the Windows Server 2003 version, insert the operating system CD and follow these steps: 1. Open your Control Panel and select Add or Remove Programs. 2. Click Add/Remove Windows Components. 3. Select Management and Monitoring Tools and click Detail. The Management and Monitoring Tools window opens (Figure 14-33).
Figure 14-33 Network Monitor Management and Monitoring Tools window
4. Select Network Monitor Tools and click OK. 5. Click Next to install the tool. 6. Click Finish. Network Monitor is now installed and ready for use without rebooting.
512
Tuning IBM System x Servers for Performance
14.3.2 Using Network Monitor You can start Network Monitor by clicking Start → Administrative Tools → Network Monitor. When you start Network Monitor the first time, you are presented with a window similar to that shown in Figure 14-34 where you select which adapter you want to monitor. The left part of the window displays the available adapter to monitor, and the right panel displays the properties of the selected adapter.
Figure 14-34 Select network connection
Select one of the connections and click OK to open the main window, as shown in Figure 14-35.
Figure 14-35 Network Monitor main window
Chapter 14. Windows tools
513
When the window first opens, the data capture process is stopped. Normally, you set filters to limit the data that the tool captures. There are two types of filters: Capture filters enable you to specify which types of packets are captured (for example, all HTTP traffic). Display filters capture all the frames that are running over the monitored NIC and the packets get filtered at the time of analysis. Using display filters enables the administrator to have all the data available at the time of analysis, but it generates a large log file. So, the server might run out of available space to save the information.
Configuring filters You can configure filters by clicking Capture → Filter or by pressing F8. The window that is shown in Figure 14-36 opens.
Figure 14-36 Capture Filter main window
514
Tuning IBM System x Servers for Performance
Creating a new filter or loading a filter In this window, you can create a new filter or load a filter that you previously saved. The current filter is displayed as a logic tree. You can modify the filter by specifying the following filter options: SAP/ETYPE Double-click this line to specify protocols that you want to capture in frames. AND (Address Pairs) Double-click this line to filter on computer address pairs. Exclude statements take logical precedence over Include statements. Regardless of the sequence in which the statements appear, EXCLUDE statements get evaluated first. AND (Pattern Matches) Double-click this line to capture only frames that include a specified pattern at a specified offset. To save the current filter to a file, click Save. To retrieve a previously saved filter, click Load. When you have completed the filter, click OK to return to the main window. Note: After you click OK, Network Monitor might rearrange your filter to optimize the logic. As a result, the filter might look different when you next open it.
Starting the capture process There are three ways to start capturing network traffic: Press F10. Click Capture → Start. Click the Play button (
) in the toolbar.
Chapter 14. Windows tools
515
Figure 14-37 illustrates the capture process.
Figure 14-37 Starting the capture
Stopping the capture process After you have captured the data that you want, stop the capture process. There are two options when stopping a capture: You can stop the capture process and view the result by performing any of the following tasks: – Press Shift+F11. – Click Capture → Stop and View. – Click the Stop and View icon (
).
You can stop the capture and not display the data that was captured by performing any of the following tasks: – Press F11. – Click Capture → Stop. – Click the Stop icon (
516
).
Tuning IBM System x Servers for Performance
Viewing a capture summary You can subsequently view the data by performing any of the following tasks: Press F12. Click Capture → Display Captured Data. Click the View icon (
).
The window that shows the captured data looks similar to that shown in Figure 14-38. The data in this window is unfiltered, so your first step might be to click Display → Filter to filter the data. Configuring this filter is the same process as in “Creating a new filter or loading a filter” on page 515.
Figure 14-38 Capture summary
Chapter 14. Windows tools
517
After you have located a packet that is important for your analysis, double-click it. Then, you get a more detailed view of the packet, as shown in Figure 14-39.
Figure 14-39 Selected captured packet
This window is divided into three sections: The first section is a duplication of the previous window, with the selected data point highlighted. If you want to view another packet in the detailed view you can select it here. The second section includes the packet’s content in a decoded tree format. Our example is that of an ICMP PING packet and Figure 14-40 shows more detail about the packet’s contents.
Figure 14-40 Expanded ICMP packet
518
Tuning IBM System x Servers for Performance
The third section includes the raw data of the packet in hexadecimal format. The column to the far right includes the reprint of the data in ASCII text. So, for example, if a password or an e-mail is delivered over your network and it is not encrypted, you can read it here. To illustrate this example, we transferred a small text file over a non-encrypted Ethernet connection and captured the packets (Figure 14-41). In the right-side of the third pane, you can see the clear text.
Figure 14-41 Clear text can be read by Network Monitor
Network Monitor tips In this redbook, we show the main steps so that you can become with this tool. If an administrator plans the integration of Network Monitor in the infrastructure and customizes the right filters, Network Monitor is a powerful tool to keep the network running smoothly, and the network administrator can identify patterns to prevent or solve network problems. Tips for running Network Monitor: Capture only the minimum amount of network statistics. Use capture filters to capture only the needed packets so as to keep the log file small. This will makes a quick diagnostic easier. Use display filters. When viewing the captured data, use display filters even if you have used filters while capturing the data..
Chapter 14. Windows tools
519
Run Network Monitor during low usage time. If you run the tool, use it during time of low-usage or only for short periods of time to make the analysis more clear and to decrease the effect that it has on your production workload. For more detailed information about Network Monitor, go to: http://msdn.microsoft.com/library/en-us/netmon/netmon/ network_monitor.asp
14.4 Other Windows tools Since Windows 2000, Microsoft has shipped a set of support tools with its server operating systems. These support tools are not installed by default, but you can install them from the installation CD-ROM. Microsoft produces a Resource Kit for both Windows 2000 Server and Windows Server 2003, and we recommend that you install and use it. At the time of writing, the Windows Server 2003 Resource Kit was available for download at no charge from: http://www.microsoft.com/windowsServer2003/downloads/tools/default.mspx Many of the tools that were in the Windows 2000 support tools or Resource Kit are included in the standard Windows Server 2003 build. For example, the typeperf command that was part of the Windows 2000 resource kit is now included as standard in Windows Server 2003. Table 14-6 lists a number of these tools and provides the executable, where the tool is installed, and a brief description. Table 14-6 Windows Server 2003 performance tools
Name
Executable
Location
Description
Clear memory
clearmem.exe
Resource Kit
Command-line tool. Used to clear RAM pages.
Logman
logman.exe
Windows
Command-line tool to manager performance monitor.
Memory consumer tool
consume.exe
Resource Kit
Command-line tool that will consume: memory pagefile disk space CPU time kernel pool
Empty working set
empty.exe
Resource Kit
Frees the working set of a specified task or process.
520
Tuning IBM System x Servers for Performance
Name
Executable
Location
Description
Extensible performance counter list
exctrlst.exe
Support Tools
Displays all the object and counter information about local or remote computer.
Device Console Utility
devcon.exe
Support Tools
Command-line tool used to enable, disable or display detailed information for installed devices.
Defrag
defrag.exe
Windows
Used to defragment hard disks.
Page fault monitor
pfmon.exe
Resource Kit
Allows the monitoring of page faults while a specified application is running.
Process Monitor
pmon.exe
Resource Kit
Displays processes and their relevant CPU time, page faults and memory in a console window.
Process Viewer
pviewer.exe
Support Tools
GUI version on Process Monitor.
Performance data block dump utility
showperf.exe
Resource Kit
GUI tool to load performance counter DLL on a local or remote server. Will then collect and display the data from the counter.
Typeperf
typeperf.exe
Windows
Displays performance counter data at a console.
Tracerpt
tracerpt.exe
Windows
Used to process trace logs.
Windows Program Timer
ntimer.exe
Resource Kit
Will return the length of time a particular application runs for.
Virtual Address Dump
vadump.exe
Resource Kit
Command-line tool that will display virtual address space.
Tip: With Windows Server 2003, you can schedule the defragmentation of your local hard disks using a combination of the task scheduler and defrag.exe. However, you might want to invest in a third-party tool for the enhanced features that they offer. See 11.17.7, “Use disk defragmentation tools regularly” on page 361 for details.
Chapter 14. Windows tools
521
14.5 Windows Management Instrumentation Up until now, we have centered our discussion of performance tools largely on the Performance console. The Performance console is a fantastic tool for monitoring one or possibly a few servers. However, when you want to monitor perhaps 10 or more servers, it become laborious to set up all those servers. One alternative is to write a shell script using some of the command line tools listed in Table 14-6 on page 520 to obtain the data. A better alternative is to access the data directly using Windows Management Instrumentation (WMI). WMI uses industry standards to retrieve management and event monitoring data from systems in the network. WMI effectively provides a large collection of data available to applications such as those written in VBScript or C++ (some functions are not available in VBScript). In this section, we describe a script that we have created in VBScript that calls WMI to retrieve and to display data from multiple systems. Tip: WMI was introduced in Windows 2000 Server. However, Microsoft has enhanced WMI significantly between Windows 2000 Server and Windows Server 2003 and some objects that are available in Windows Server 2003 might not be available in Windows 2000 Server. For our purposes, we are interested in two particular classes in the WMI repository: Win32_PerfRawData The Win32_PerfRawData class and its subclasses were introduced with Windows 2000 and provide access to the raw data that is made available to the Performance console. Win32_PerfFormattedData The Win32_PerfFormattedData class and subclasses were introduced with Windows Server 2003 and include processed data. This data is the same as displayed by the Performance console. For our purposes, the Win32_PerfRawData class suffices because it runs off both Windows 2000 and Windows Server 2003 computers. Win32_PerRawData has a number of subclasses, and these hold the data in which we are interested. Table 14-7 shows some of the more commonly used Performance console objects and their equivalent classes in WMI. These are not the only performance classes that are available in WMI.
522
Tuning IBM System x Servers for Performance
Table 14-7 Performance console objects and their equivalent WMI classes
Performance console object
Class
Cache
Win32_PerfRawData_PerfOS_Cache
LogicalDisk
Win32_PerfRawData_PerfDisk_LogicalDisk
Memory
Win32_PerfRawData_PerfOS_Memory
Paging File
Win32_PerfRawData_PerfOS_PagingFile
PhysicalDisk
Win32_PerfRawData_PerfDisk_PhysicalDisk
Process
Win32_PerfRawData_PerfProc_Process
Processor
Win32_PerfRawData_PerfOS_Processor
Server
Win32_PerfRawData_PerfNet_Server
Example 14-2 illustrates our example scenario. We have a large number of servers and want to monitor the memory usage of each. We want to retrieve this data and record it in a file for review. Example 14-2 shows a VBScript that collects data from multiple servers and then outputs it to the console. Example 14-2 Perfdata.vbs: collecting performance data using WMI
‘ ‘ ‘ ‘ ‘ ‘ ‘ ‘
************************************************************************* ***** IBM ITSO **** ************************************************************************* Script Name = perdata.vbs Version = 1.0 Author = Brian Jeffery Description = Collect performance data from multiple servers
‘ ‘ ‘ ‘ ‘ ‘ ‘
************************************************************************* History ************************************************************************* Date Version Description ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 19/07/2004 1.0 Initial Release *************************************************************************
Option Explicit ‘ **** Define Global Constants Const FORREADING = 1
Chapter 14. Windows tools
523
‘ **** Declare Dim oArgs Dim oFS Dim oInStream
Objects ‘Wscript.Arguments object ‘Scripting.FileSystemObject ‘Stream with text input from file
‘ **** Declare Variables Dim sServerListFile ‘name of server file Dim sServer ‘individual server to connect to
‘ **** Initialise Objects Set oFS = CreateObject(“Scripting.FileSystemObject”) Set oArgs = Wscript.Arguments
‘ **** Ensure Cscript.exe used to run script If (LCase(Right(WScript.FullName, 11)) = “wscript.exe”) Then Usage(“Script must be run using cscript.exe”) End If ‘ **** populate variables with values from argument If oArgs.Count <> 1 Then ‘If the number of arguments not equal to 1 exit gracefully Usage(“Invalid number of arguments supplied”) Else ‘Arguments = 1 then set the sServerListFile variable to its value sServerListFile = oArgs(0) End If ‘ **** open server list If oFS.FileExists(sServerListFile) Then ‘ file found. Now open it Set oInstream = oFS.OpenTextFile(sServerListFile, FORREADING) Else ‘text file not found, display usage and exit Usage(“Unable to open “ & sServerListFile) End If ‘ **** loop through each line of the text file. Each line should ‘ correspond to a text file. Do Until oInStream.AtEndOfStream sServer = Trim(oInstream.ReadLine) ‘Remove any leading backslashes If InStr(sServer, “\\”) = 1 Then
524
Tuning IBM System x Servers for Performance
sServer = Mid(sServer, 3) End If ‘run sub routine collectdata, supplying name of server CollectData(sServer) Loop ‘ ************************************************************************* ‘ End of Main script ‘ ************************************************************************* ‘ ************************************************************************* ‘ Sub CollectData ‘ ************************************************************************* Sub CollectData(p_sComputerName) ‘ **** Declare Dim oWMI Dim colItems Dim oItem
objects ‘Ref to WinMgmts ‘Collection of objects returned by the query ‘Individual object from the collection
‘ **** Declare Variables Dim sQuery Dim sConnect ‘ **** connect to WinMgmts on remote server Set oWMI = GetObject(“WinMgmts:{authenticationLevel=pkt}!\\”_ & p_sComputerName) ‘ **** set WMI query sQuery = “SELECT * FROM Win32_PerfRawData_PerfOS_Memory” ‘ **** Execute WMI Query Set colItems = oWMI.ExecQuery(sQuery) ‘ **** Now Display the results For Each oItem in colItems Wscript.Echo p_sComputerName Wscript.Echo vbTab & “Available Memory = “ & _ oItem.AvailableMBytes & “ MB” Wscript.Echo vbTab & “Commit Limit = “ & _ Round(oItem.CommitLimit/1024^2, 2) & “ MB” Wscript.Echo vbTab & “Committed MB
= “ & _
Chapter 14. Windows tools
525
Round(oItem.CommittedBytes/1024^2, 2) & “ MB” Wscript.Echo Next End Sub ‘ ************************************************************************* ‘ End Sub CollectData ‘ ************************************************************************* ‘ ************************************************************************* ‘ Sub Usage ‘ *************************************************************************
Sub Usage(p_sError) Dim sText sText sText sText sText sText sText
= = = = = =
“Perfdata.vbs failed to run because:” & vbCrLf & vbCrLf sText & vbTab & p_sError & vbCrLf & vbCrLf sText & “Usage:” & vbCrLf & vbCrLf sText & vbTab & “cscript /nologo perfdata.vbs ” sText & vbCrLf & vbCrLf & “Where:” & vbCrLf sText & vbTab & “ refers to the file with list of “_ & “servers” Wscript.Echo sText ‘ exit the script Wscript.Quit End Sub ‘ ************************************************************************* ‘ End Sub Usage ‘ *************************************************************************
526
Tuning IBM System x Servers for Performance
The steps to use this script are as follows: 1. Create a file, perfdata.vbs, and paste the code shown in Example 14-2 on page 523 into it. 2. Create a text file and enter the names of each of your servers. (See Example 14-3 on page 527) for a sample. Save this file in the same folder as the script file. Note: The script is simple and assumes the file name of this text file has no spaces in it. Example 14-3 A possible list of servers, saved as servers.txt
chaumes paws gbrhmx097 3. Run the script. Assuming that you saved the list of servers as servers.txt, you would run the script as follows: cscript.exe /nologo perfdata.vbs servers.txt 4. Assuming that everything runs correctly, you should get an output similar to that shown in Example 14-4. Example 14-4 Example output of perfdata.vbs
chaumes Available Memory = 48 MB Commit Limit = 495.23 MB Committed MB = 68.02 MB paws Available Memory = 69 MB Commit Limit = 558.77 MB Committed MB = 126.07 MB gbrhmx097 Available Memory = 105 MB Commit Limit = 1694.14 MB Committed MB = 820.14 MB Each server is listed in turn with three pieces of data: Available Memory: lists the available physical RAM Commit Limit: the size of the paging file Committed MB: the actual amount of the paging file that is used
Chapter 14. Windows tools
527
To output this data to a file, simply pipe the output to a text file through the command line, as in: cscript /nologo perfdata.vbs servers.txt >output.txt If you want the script to append existing data rather than overwrite then simply replace the > with a >>. Tip: If you want any possible errors to be saved to the output text file along with the data, then add 2>&1 to the end of the command string: cscript /nologo perfdata.vbs servers.tx >output.txt 2>&1 For more information about WMI and performance data, see: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wmisdk /wmi/performance_counter_classes.asp Note: We have not included a detailed description of perfdata.vbs, because the comments in the script should be sufficient for anyone who is familiar with VBScript and WMI. However, if you are not familiar with either WMI or VBScript and you want to know more, we recommend Windows Management Instrumentation (WMI), New Riders, by Matthew Lavy and Ashley Meggitt, ISBN: 1578702607.
14.6 VTune VTune is a software tool from Intel that helps your analyze your system and applications. It works across the range of Intel architectures to detect hotspots, areas in your code that take long time to execute. VTune collects performance data on your system and displays the result in a graphical surface to help you determine the hotspots and what is causing them. It also helps you decide how to eliminate them. VTune allows you to track things, such as where the processor is spending its time, where misaligned memory references occur, or where branch mispredictions happen. VTune is a utility that is available for Windows or for Linux operating systems. You can obtain a 30-day trial version of the latest VTune software from Intel at: https://registrationcenter.intel.com/EvalCenter/EvalForm.aspx?ProductID=585 The hardware and software requirements for using this tool are listed at: http://www.intel.com/cd/software/products/asmo-na/eng/220001.htm
528
Tuning IBM System x Servers for Performance
If you install the software, it opens a Web site on your server automatically, with a detailed introduction about how to use this tool, as shown in Figure 14-42. If the site does not opened automatically, you can find it under Start → All Programs → Intel VTune Performance Analyzer → Getting Started Tutorial.
Figure 14-42 VTune introduction
Chapter 14. Windows tools
529
When you run VTune, it collects data and displays it in an graphical interface. Figure 14-43 shows the standard collection view. Each line in the chart represents data in a specific performance counter.
Figure 14-43 VTune sample collection
530
Tuning IBM System x Servers for Performance
The counters are listed in the lower pane. You can highlight a counter in the chart by double clicking it, and you can get an explanation of the purpose of the counter by right clicking, as shown in Figure 14-44.
Figure 14-44 Counter explanation
Chapter 14. Windows tools
531
You can also mark a part of the collected data, as shown in Figure 14-45, to see an analysis of the data as shown in Figure 14-46 on page 533.
Figure 14-45 Marked area
532
Tuning IBM System x Servers for Performance
Figure 14-46 Tuning analysis
Chapter 14. Windows tools
533
Other views are also available. A summary view shows the data for each counter represented as a bar diagram. The summary view shows the minimum, maximum, and average value of your counter, as shown in Figure 14-47.
Figure 14-47 Summary view
To analyze hotspots and bottlenecks with this tool, it is important that you understand what each counter means. Read the explanation of each counter like the one shown in Figure 14-44 on page 531 if you are unsure. In addition, the introduction (Figure 14-42 on page 529) includes some animated examples to better understand how to analyze the data.
534
Tuning IBM System x Servers for Performance
You can use VTune to record server behavior over time under real-life conditions. For example, you can take samples at the same time each week, at different times of the day, or before and after making changes. Doing so, you can quickly recognize trends, identify workload peaks, and avoid performance problems. If you need further information, we recommend that you to download the evaluation version of VTune and read the documentation that is available with that version. You can obtain a 30-day trial version of the latest VTune software from Intel at: https://registrationcenter.intel.com/EvalCenter/EvalForm.aspx?ProductID=585 The evaluation version includes some animated examples that provide an overview of how to use this tool.
Chapter 14. Windows tools
535
536
Tuning IBM System x Servers for Performance
15
Chapter 15.
Linux tools Various tools are available with which you can monitor and analyze the performance of your server. Most of these tools use existing information that is stored in the /proc directory to present it in a readable format. This chapter includes a list of useful command line and graphical tools that are available as packages in Red Hat Enterprise Linux AS or SUSE Linux Enterprise Server or that you can download from the Internet. Three utilities—sar, iostat, and mpstat—are part of the Sysstat package that is provided on the distribution CD-ROMs or that is available from the Sysstat home page: http://perso.wanadoo.fr/sebastien.godard/
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.
537
Table 15-1 lists the tools that we discuss in this chapter. Table 15-1 Linux performance monitoring tools
Tool
Most useful tool function
Page
uptime
Average system load
539
dmesg
Hardware and system information
540
top
Processor activity
541
iostat
Average CPU load and disk activity
543
vmstat
System activity
545
sar
Collect and report system activity
546
KDE System Guard
Real time systems reporting and graphing
547
free
Memory usage
554
traffic-vis
Network monitoring (SUSE Linux Enterprise Server only)
554
pmap
Process memory usage
557
strace
Programs
558
ulimit
System limits
559
mpstat
Multiprocessor usage
560
xPL
The System x Performance Logger for Linux (also known as PLPerf)
561
nmon
IBM-developed tool showing data from the /proc file system
571
These tools are in addition to the Capacity Manager tool, which is part of IBM Director. The Capacity Manager tool monitors system performance over a period of time. IBM Director can be used on different operating system platforms. It makes it easier to collect and to analyze data in a heterogeneous environment. For more information about Capacity Manager, see Chapter 17, “Capacity Manager” on page 591.
538
Tuning IBM System x Servers for Performance
15.1 The uptime command You can use the uptime command to see how long the server has been running and how many users are logged on, as well as for a quick overview of the average load of the server. The system load average is displayed for the last 1-, 5-, and 15-minute intervals. The load average is not a percentage but is the number of processes in the queue that are waiting to be processed. If the processes that request CPU time are blocked (which means that the CPU has no time to process them), the load average increases. Alternatively, if each process gets immediate access to CPU time and there are no CPU cycles lost, the load decreases. The optimal value of the load is 1, which means that each process has immediate access to the CPU and there are no CPU cycles lost. The typical load can vary from system to system. For a uniprocessor workstation, 1 or 2 might be acceptable, while you might see values of 8 to 10 on multiprocessor servers. You can use uptime to pinpoint an issue with your server or the network. If, for example, a network application is running poorly, you can run uptime, and you will see whether the system load is high. If the system load is not high, the problem might be related to your network rather than to your server. Tip: You can also use w instead of uptime, which also provides information about who is logged on to the machine and what the user is doing. Example 15-1 Sample output of uptime
1:57am
up 4 days 17:05,
2 users,
load average: 0.00, 0.00, 0.00
Chapter 15. Linux tools
539
15.2 The dmesg command The main purpose of the dmesg command is to display kernel messages. The dmesg command can provide helpful information in case of hardware issues or issues with loading a module into the kernel. Example 15-2 shows partial output from the dmesg command. In addition, with dmesg, you can determine what hardware is installed in your server. During every boot, Linux checks your hardware and logs information about it. You can view these logs using the command /bin/dmesg. Example 15-2 Partial output from dmesg
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,1), internal journal EXT3-fs: mounted filesystem with ordered data mode. IA-32 Microcode Update Driver: v1.11 ip_tables: (C) 2000-2002 Netfilter core team 3c59x: Donald Becker and others. www.scyld.com/network/vortex.html See Documentation/networking/vortex.txt 01:02.0: 3Com PCI 3c980C Python-T at 0x2080. Vers LK1.1.18-ac 00:01:02:75:99:60, IRQ 15 product code 4550 rev 00.14 date 07-23-00 Internal config register is 3800000, transceivers 0xa. 8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface. MII transceiver found at address 24, status 782d. Enabling bus-master transmits and whole-frame receives. 01:02.0: scatter/gather enabled. h/w checksums enabled divert: allocating divert_blk for eth0 ip_tables: (C) 2000-2002 Netfilter core team Intel(R) PRO/100 Network Driver - version 2.3.30-k1 Copyright (c) 2003 Intel Corporation divert: allocating divert_blk for eth1 e100: selftest OK. e100: eth1: Intel(R) PRO/100 Network Connection Hardware receive checksums enabled cpu cycle saver enabled ide-floppy driver 0.99.newide hda: attached ide-cdrom driver. hda: ATAPI 48X CD-ROM drive, 120kB Cache, (U)DMA Uniform CD-ROM driver Revision: 3.12 Attached scsi generic sg4 at scsi1, channel 0, id 8, lun 0,
540
Tuning IBM System x Servers for Performance
type 3
15.3 The top command The top command shows actual processor activity. By default, it displays the most CPU-intensive tasks that are running on the server and updates the list every five seconds. You can sort the processes by processor ID (numerically), age (newest first), time (cumulative time), and resident memory usage and time (time the process has occupied the CPU since startup). Example 15-3 Example output from top command
top - 02:06:59 up 4 days, 17:14, 2 users, load Tasks: 62 total, 1 running, 61 sleeping, 0 Cpu(s): 0.2% us, 0.3% sy, 0.0% ni, 97.8% id, Mem: 515144k total, 317624k used, 197520k Swap: 1048120k total, 12k used, 1048108k PID 13737 238 1 2 3 4 5 6 7 8 9 10 13 14 16 17 18
USER root root root root root root root root root root root root root root root root root
PR 17 5 16 RT 34 RT 34 5 5 5 5 15 5 16 15 13 13
NI 0 -10 0 0 19 0 19 -10 -10 -10 -10 0 -10 0 0 -10 -10
VIRT 1760 0 588 0 0 0 0 0 0 0 0 0 0 0 0 0 0
average: 0.00, 0.00, 0.00 stopped, 0 zombie 1.7% wa, 0.0% hi, 0.0% si free, 66068k buffers free, 179632k cached
RES SHR S %CPU %MEM 896 1540 R 0.7 0.2 0 0 S 0.3 0.0 240 444 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0 0 0 S 0.0 0.0
TIME+ 0:00.05 0:01.56 0:05.70 0:00.00 0:00.00 0:00.00 0:00.00 0:00.02 0:00.00 0:00.09 0:00.01 0:00.00 0:00.02 0:00.45 0:00.61 0:00.00 0:00.00
COMMAND top reiserfs/0 init migration/0 ksoftirqd/0 migration/1 ksoftirqd/1 events/0 events/1 kblockd/0 kblockd/1 kirqd khelper/0 pdflush kswapd0 aio/0 aio/1
You can further modify the processes using renice to give a new priority to each process. If a process hangs or occupies too much CPU, you can kill the process (using the kill command). The columns in the output are as follows: PID
Process identification.
USER
Name of the user who owns (and perhaps started) the process.
PRI
Priority of the process (see 15.3.1, “Process priority and nice levels” on page 542 for details).
Chapter 15. Linux tools
541
NI
Niceness level (that is, whether the process tries to be nice by adjusting the priority by the number given) See 15.3.1, “Process priority and nice levels” on page 542 for more information.
SIZE
Amount of memory (code+data+stack) in KB that are being used by the process.
RSS
Amount of physical RAM used in KB.
SHARE
Amount of memory shared with other processes in KB.
STAT
State of the process: S=sleeping, R=running, T=stopped or traced, D=interruptible sleep, Z=zombie. Zombie processes are discussed further in 15.3.2, “Zombie processes” on page 543.
%CPU
Share of the CPU usage (since the last screen update).
%MEM
Share of physical memory.
TIME
Total CPU time that is used by the process since it was started.
COMMAND
Command line used to start the task (including parameters).
Tip: The /bin/ps command gives a snapshot view of the current processes.
15.3.1 Process priority and nice levels Process priority is a number that determines the order in which the process is handled by the CPU. The kernel adjusts this number up and down as needed. The nice value is a limit on the priority. The priority number is not allowed to go below the nice value (a lower nice value is a more favored priority). It is not possible to change the priority of a process. This is only indirectly possible through the use of the nice level of the process. Note that it might not always be possible to change the priority of a process using the nice level. If a process is running too slowly, you can assign more CPU to it by giving it a lower nice level. Of course, doing so means that all other programs will have fewer processor cycles and will run more slowly. Linux supports nice levels from 19 (lowest priority) to -20 (highest priority). The default value is zero (0). To change the nice level of a program to a negative number (which makes it a higher priority process), it is necessary to log on or su to root. For example, to start the program xyz with a nice level of -5, issue the command: nice -n -5 xyz
542
Tuning IBM System x Servers for Performance
To change the nice level of a program already running, issue the command: renice level pid If we wanted to change the priority of the xyz program that has a PID of 2500 to a nice level of 10, issue the following command: renice 10 2500
15.3.2 Zombie processes When a process terminates, having received a signal to do so, it normally takes some time to finish all tasks (such as closing open files) before the process terminates itself. In that normally very short time frame, the process is called a zombie process. After the process has completed all the shutdown tasks, it reports to the parent process that it is about to terminate. Sometimes, a zombie process is unable to terminate itself, in which case you will see that it has a status of Z (zombie). It is not possible to kill such a process with the kill command, because it is already considered dead. If you cannot kill a zombie process, you can kill the parent process and then the zombie disappears as well. However, if the parent process is the init process, you should not kill it. The init process is a very important process. Therefore, you might need to reboot to get rid of the zombie process.
15.4 The iostat command The iostat utility is part of Red Hat Enterprise Linux AS distribution. The iostat command is also part of the Sysstat set of utilities, which are available from: http://perso.wanadoo.fr/sebastien.godard/ The iostat command lets you see the average CPU times since the system was started, in a way similar to uptime. In addition, however, iostat creates a report about the activities of the disk subsystem of the server. The report has two parts: CPU utilization and device (disk) utilization. To use iostat to perform detailed I/O bottleneck and performance tuning, see 20.4.1, “Finding bottlenecks in the disk subsystem” on page 698. Example 15-4 shows sample output for the iostat command.
Chapter 15. Linux tools
543
Example 15-4 Sample output of iostat
Linux 2.4.21-9.0.3.EL (x232) avg-cpu:
Device: dev2-0 dev8-0 dev8-1 dev8-2 dev8-3
%user 0.03
%nice 0.00 tps 0.00 0.45 0.00 0.00 0.00
05/11/2004
%sys 0.02
%idle 99.95
Blk_read/s 0.00 2.18 0.00 0.00 0.00
Blk_wrtn/s 0.04 2.21 0.00 0.00 0.00
Blk_read 203 166464 16 8 344
Blk_wrtn 2880 168268 0 0 0
The CPU utilization report has four sections: %user
Shows the percentage of CPU utilization that was taken up while executing at the user level (applications).
%nice
Shows the percentage of CPU utilization that was taken up while executing at the user level with a nice priority. (For more information about priority and nice levels, see 15.3.1, “Process priority and nice levels” on page 542.)
%sys
Shows the percentage of CPU utilization that was taken up while executing at the system level (kernel).
%idle
Shows the percentage of time the CPU was idle.
The device utilization report is split into the following sections: Device: The name of the block device. tps: The number of transfers per second (I/O requests per second) to the device. Multiple single I/O requests can be combined in a transfer request, because a transfer request can have different sizes. Blk_read/s, Blk_wrtn/s: Blocks read and written per second indicate data read/written from/to the device in seconds. Blocks can also have different sizes. Typical sizes are 1024, 2048 or 4048 bytes, depending on the partition size. For example, the block size of /dev/sda1 can be found with: dumpe2fs -h /dev/sda1 |grep -F "Block size" which gives an output similar to: dumpe2fs 1.34 (25-Jul-2003) Block size: 1024 Blk_read, Blk_wrtn: This indicates the total number of blocks read/written since the boot.
544
Tuning IBM System x Servers for Performance
15.5 The vmstat command The vmstat command provides information about processes, memory, paging, block I/O, traps, and CPU activity. Example 15-5 Example output from vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 0 154804 77328 910900 0 0 4 6 103 19 0 0 100 0 The columns in the output are as follows: Process r b
The number of processes waiting for runtime. The number of processes in uninterruptable sleep.
Memory swpd The amount of virtual memory used (KB). free The amount of idle memory (KB). buff The amount of memory used as buffers (KB). Swap si so
Amount of memory swapped from the disk (KBps). Amount of memory swapped to the disk (KBps).
I/O bi bo
Blocks sent to a block device (blocks/s). Blocks received from a block device (blocks/s).
System in cs
The number of interrupts per second, including the clock. The number of context switches per second.
CPU (these are percentages of total CPU time) us sy id wa
Time spent running non-kernel code (user time, including nice time). Time spent running kernel code (system time). Time spent idle. Prior to Linux 2.5.41, this included IO-wait time. Time spent waiting for IO. Prior to Linux 2.5.41, this appeared as zero.
Chapter 15. Linux tools
545
15.6 The sar command The sar utility is part of Red Hat Enterprise Linux AS distribution. The sar command is also part of the Sysstat set of utilities, which are available from: http://perso.wanadoo.fr/sebastien.godard/ The sar command is used to collect, report, or save system activity information. The sar command consists of three applications: sar which displays the data, and sa1 and sa2 which are used for collecting and storing the data. By using sa1 and sa2, you can configure the system to get the information and log it for later analysis. To do this, you must configure a cron job by adding the lines shown in Example 15-6 to the /etc/crontab file. Example 15-6 Example of starting automatic log reporting with cron
# 8am-7pm activity reports every 10 minutes during weekdays. */10 8-18 * * 1-5 /usr/lib/sa/sa1 600 6 & # 7pm-8am activity reports every an hour during weekdays. 0 19-7 * * 1-5 /usr/lib/sa/sa1 & # Activity reports every an hour on Saturday and Sunday. 0 * * * 0,6 /usr/lib/sa/sa1 & # Daily summary prepared at 19:05 5 19 * * * /usr/lib/sa/sa2 -A & Alternatively, you can use sar to run almost real-time reporting from the command line, as shown in Example 15-7 on page 546. From the collected data, you get a detailed overview of your CPU utilization (%user, %nice, %system, %idle), memory paging, network I/O, and transfer statistics, process creation activity, activity for block devices, and interrupts/second over time. Example 15-7 Ad hoc CPU monitoring
[root@x232 root]# sar -u 3 10 Linux 2.4.21-9.0.3.EL (x232) 02:10:40 02:10:43 02:10:46 02:10:49 02:10:52 02:10:55 02:10:58 02:11:01
546
PM PM PM PM PM PM PM PM
CPU all all all all all all all
%user 0.00 0.33 0.00 7.14 71.43 0.00 0.00
Tuning IBM System x Servers for Performance
05/22/2004 %nice 0.00 0.00 0.00 0.00 0.00 0.00 0.00
%system 0.00 0.00 0.00 18.57 28.57 100.00 0.00
%idle 100.00 99.67 100.00 74.29 0.00 0.00 0.00
02:11:04 PM 02:11:07 PM 02:11:10 PM Average:
all all all all
0.00 50.00 0.00 1.62
0.00 0.00 0.00 0.00
100.00 50.00 100.00 3.33
0.00 0.00 0.00 95.06
15.7 KDE System Guard KDE System Guard (KSysguard) is the KDE task manager and performance monitor. It features a client/server architecture that enables monitoring of local as well as remote hosts. The graphical front end (as shown in Figure 15-1) uses sensors to retrieve the information that it displays. A sensor can return simple values or more complex information such as tables. For each type of information, one or more displays are provided. Displays are organized into worksheets that can be saved and loaded independently of each other.
Figure 15-1 Default KDE System Guard window
Chapter 15. Linux tools
547
The KSysguard main window (Figure 15-1) consists of a menu bar, an optional tool bar and status bar, the sensor browser, and the work space. When first started, you see your local machine listed as localhost in the sensor browser and two tabs in the work space area. This is the default setup. Each sensor monitors a certain system value. You can drag and drop all of the displayed sensors in the work space. There are three options: You can delete and replace sensors in the actual work space. You can edit work sheet properties and increase the number of row and or columns. You can create a new worksheet and drop new sensors meeting your needs.
15.7.1 The KSysguard work space By looking at the work space in Figure 15-2, you notice that there are two tabs: System Load, the default view when first starting up KSysguard Process Table
Figure 15-2 KDE System Guard sensor browser
548
Tuning IBM System x Servers for Performance
System Load The System Load worksheet consists of four sensor windows:
CPU Load Load Average (1 Minute) Physical Memory Swap Memory
You will note from the Physical Memory window that it is possible to have multiple sensors displayed within one window. To determine which sensors are being monitored in a given window, mouse over the graph and some descriptive text appears. Another way to do this is to right-click the graph and click Properties, then go to the Sensors tab, as shown in Figure 15-3. The Sensors tab also shows a key of what each color represents on the graph.
Figure 15-3 Sensor Information, Physical Memory Signal Plotter
Chapter 15. Linux tools
549
Process Table The Process Table tab displays information about all the running processes on the server (Figure 15-4). The table, by default, is sorted by System CPU utilization. You can change the way the table is sorted by clicking the heading by which you want to sort.
Figure 15-4 Process Table view
550
Tuning IBM System x Servers for Performance
Configuring a work sheet For your environment or the particular area that you want to monitor, it might be necessary to use different sensors for monitoring. The best way to do this is to create a custom work sheet. In this section, we guide you through the steps that are required to create the work sheet that is shown in Figure 15-7 on page 553. The steps to create a worksheet are as follows: 1. Create a blank worksheet by clicking File → New to open the window that is shown in Figure 15-5.
Figure 15-5 Properties for new worksheet
2. Enter a title and a number of rows and columns. This gives you the maximum number of monitor windows, which in our case is four. When the information is complete, click OK to create the blank worksheet, as shown in Figure 15-6. Note: The fastest update interval that can be defined is two seconds.
Chapter 15. Linux tools
551
Figure 15-6 Empty worksheet
3. Now, you can complete the sensor boxes by simply dragging the sensors on the left side of the window to the desired box on the right. The display choices are: – Signal Plotter. This sensor style displays samples of one or more sensors over time. If several sensors are displayed, the values are layered in different colors. If the display is large enough, a grid is displayed to show the range of the plotted samples. By default, the automatic range mode is active, so the minimum and maximum values are set automatically. If you want fixed minimum and maximum values, you can de-activate the automatic range mode and set the values in the Scales tab from the Properties dialog window (which you access by right-clicking the graph). – Multimeter. The Multimeter displays the sensor values as a digital meter. In the properties dialog, you can specify a lower and upper limit. If the range is exceeded, the display is colored in the alarm color.
552
Tuning IBM System x Servers for Performance
– BarGraph. The BarGraph displays the sensor value as dancing bars. In the properties dialog, you can also specify the minimum and maximum values of the range and a lower and upper limit. If the range is exceeded, the display is colored in the alarm color. – Sensor Logger: The Sensor Logger does not display any values, but logs them in a file with additional date and time information. For each sensor, you have to define a target log file, the time interval the sensor will be logged and whether alarms are enabled. 4. Click File → Save to save the changes to the worksheet. Note: When you save a work sheet, it is saved in your home directory, which might prevent other administrators from using your custom worksheet.
Figure 15-7 Example worksheet
You can find more information about KDE System Guard online at: http://docs.kde.org/en/3.2/kdebase/ksysgaurd
Chapter 15. Linux tools
553
15.8 The free command The /bin/free command displays information about the total amounts of free and used memory (including swap) on the system, as shown in Example 15-8. It also includes information about the buffers and cache used by the kernel. Example 15-8 Example output from the free command
total cached Mem: 1291980 772016 -/+ buffers/cache: Swap: 2040244
used
free
998940
293040
137568 0
1154412 2040244
shared
buffers 0
89356
15.9 Traffic-vis Traffic-vis is a suite of tools that determine which hosts have been communicating on an IP network, with whom they have been communicating, and the volume of communication that has taken place. The final report can be generated in plain text, HTML, or GIF. Note: Traffic-vis is for SUSE Linux Enterprise Server only. Start the program to collect data on interface eth0, for example: traffic-collector -i eth0 -s /root/output_traffic-collector After the program starts, it is detached from the terminal and begins the collection of data. You can control the program by using the killall command to send a signal to the process. For example, to write the report to disk, issue the following command: killall -SIGUSR1 traffic-collector To stop the collection of data, issue this command: killall -SIGTERM traffic-collector Important: Do not forget to run this last command. Otherwise, your system’s performance will degrade due to a lack of memory.
554
Tuning IBM System x Servers for Performance
You can sort the output by packets, bytes, TCP connections, the total of each one, or the number of sent or received of each one. For example, to sort total packets sent and received on hosts, use this command: traffic-sort -i output_traffic-collector -o output_traffic-sort -Hp To generate a report in HTML format that displays the total bytes transferred, total packets recorded, total TCP connections requests, and other information about each server in the network, run, use this command: traffic-tohtml -i output_traffic-sort -o output_traffic-tohtml.html This output file can be displayed in a browser, as shown in Figure 15-8.
Figure 15-8 Report generated by traffic-vis
Chapter 15. Linux tools
555
To generate a report in GIF format with a width of 600 pixels and a height of 600 pixels, use the following command: traffic-togif -i output_traffic-sort -o output_traffic-togif.gif -x 600 -y 600 Figure 15-9 shows the communication between systems in the network. You can also see that some hosts talk to others, but there are servers that never talk to each other. This output is used typically to find broadcasts in the network. To see what servers are using IPX/SPX protocol in a TCP network and to separate both networks, remember that IPX is based on broadcast packets. If we need to pinpoint others types of issues, such as damaged network cards or duplicated IPs on networks, we need to use more specific tools, such as Ethereal, which is installed by default on SUSE Linux Enterprise Server.
Figure 15-9 Report generate by traffic-vis
556
Tuning IBM System x Servers for Performance
Tip: Using pipes, it is possible to produce output in one command. For example, to generate a report in HTML, run the following command: cat output_traffic-collector | traffic-sort -Hp | traffic-tohtml -o output_traffic-tohtml.html To generate a report as a GIF file, run: cat output_traffic-collector | traffic-sort -Hp | traffic-togif -o output_traffic-togif.gif -x 600 -y 600
15.10 The pmap command The pmap command reports the amount of memory that one or more processes are using. You can use this tool to determine which processes on the server are being allocated memory and whether this amount of memory is a cause of memory bottlenecks. Example 15-9 shows the result of the following command in SUSE Linux Enterprise Server: pmap -x Example 15-9 Total amount of memory cupsd process is using (SLES)
linux:~ # pmap -x 1796 1796: /usr/sbin/cupsd Address Kbytes RSS Anon Locked Mode 08048000 244 - r-x-ffffe000 4 - ------------ ------- ------- ------- ------total kB 6364 -
Mapping cupsd [ anon ]
Example 15-10 shows the result of the following command in Red Hat Enterprise Linux AS: pmap Example 15-10 Total amount of memory smbd process is using
[root@x232 root]# pmap 8359 smbd[8359] b723c000 (1224 KB) r-xp (08:02 1368219) /lib/tls/libc-2.3.2.so b736e000 (16 KB) rw-p (08:02 1368219) /lib/tls/libc-2.3.2.so mapped: 9808 KB writable/private: 1740 KB shared: 64 KB
Chapter 15. Linux tools
557
For the complete syntax of the pmap command, issue: pmap -?
15.11 The strace command The strace command intercepts and records the system calls that are called by a process, and the signals that are received by a process. This is a useful diagnostic, instructional, and debugging tool. System administrators will find it valuable for solving problems with programs. To use the command, specify the process ID (PID) to be monitored as follows: strace -p Example 15-11 shows an example of the output of strace. Example 15-11 Output of strace monitoring httpd process, for example
[root@x232 html]# strace -p 815 Process 815 attached - interrupt to quit semop(360449, 0xb73146b8, 1) = 0 poll([{fd=4, events=POLLIN}, {fd=3, events=POLLIN, revents=POLLIN}], 2, -1) = 1 accept(3, {sa_family=AF_INET, sin_port=htons(52534), sin_addr=inet_addr("9.42.171.197")}, [16]) = 13 semop(360449, 0xb73146be, 1) = 0 getsockname(13, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("9.42.171.198")}, [16]) = 0 fcntl64(13, F_GETFL) = 0x2 (flags O_RDWR) fcntl64(13, F_SETFL, O_RDWR|O_NONBLOCK) = 0 read(13, 0x8259bc8, 8000) = -1 EAGAIN (Resource temporarily unavailable) poll([{fd=13, events=POLLIN, revents=POLLIN}], 1, 300000) = 1 read(13, "GET /index.html HTTP/1.0\r\nUser-A"..., 8000) = 91 gettimeofday({1084564126, 750439}, NULL) = 0 stat64("/var/www/html/index.html", {st_mode=S_IFREG|0644, st_size=152, ...}) = 0 open("/var/www/html/index.html", O_RDONLY) = 14 mmap2(NULL, 152, PROT_READ, MAP_SHARED, 14, 0) = 0xb7052000 writev(13, [{"HTTP/1.1 200 OK\r\nDate: Fri, 14 M"..., 264}, {"\n\n RedPaper Per"..., 152}], 2) = 416 munmap(0xb7052000, 152) = 0 socket(PF_UNIX, SOCK_STREAM, 0) = 15 connect(15, {sa_family=AF_UNIX, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory) close(15) = 0<br />
<br />
558<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
For the complete syntax of the strace command, issue: strace -?<br />
<br />
15.12 The ulimit command The ulimit command is built into the bash shell and is used to provide control over the resources that are available to the shell and to the processes that are started by it on systems that allow such control. You can use the -a option to list all parameters that you can set: ulimit -a Example 15-12 Output of ulimit<br />
<br />
[root@x232 html]# ulimit -a core file size (blocks, -c) data seg size (kbytes, -d) file size (blocks, -f) max locked memory (kbytes, -l) max memory size (kbytes, -m) open files (-n) pipe size (512 bytes, -p) stack size (kbytes, -s) cpu time (seconds, -t) max user processes (-u) virtual memory (kbytes, -v)<br />
<br />
0 unlimited unlimited 4 unlimited 1024 8 10240 unlimited 7168 unlimited<br />
<br />
The -H and -S options specify the hard and soft limits that can be set for the given resource. If the soft limit is passed, the system administrator receives a warning. The hard limit is the maximum value that can be reached before the user gets the error message Out of file handles. For example, you can set a hard limit for the number of file handles and open files (-n) as follows: ulimit -Hn 4096 For the soft limit of number of file handles and open files, use: ulimit -Sn 1024 To see the hard and soft values, issue the command with a new value as follows: ulimit -Hn ulimit -Sn<br />
<br />
Chapter 15. Linux tools<br />
<br />
559<br />
<br />
You can use this command, for example, to limit Oracle users. To set it on startup, enter the follow lines, in /etc/security/limits.conf: soft nofile 4096 hard nofile 10240 In addition, for Red Hat Enterprise Linux AS, make sure that the file /etc/pam.d/system-auth has the following entry: session<br />
<br />
required<br />
<br />
/lib/security/$ISA/pam_limits.so<br />
<br />
For SUSE Linux Enterprise Server, make sure that the files /etc/pam.d/login and /etc/pam.d/sshd have the following entry: session required<br />
<br />
pam_limits.so<br />
<br />
This entry is required so that the system can enforce these limits. For the complete syntax of the ulimit command, issue: ulimit -?<br />
<br />
15.13 The mpstat command The mpstat command is part of the Sysstat set of utilities, which are available from: http://perso.wanadoo.fr/sebastien.godard/ The mpstat command is used to report the activities of each the CPUs that are available on a multiprocessor server. Global average activities among all CPUs are also reported. Example 15-13 shows example output for the mpstat command. For example, use the follow command to display three entries of global statistics among all processors at two second intervals: mpstat 2 3 Tip: You can use this command on non-SMP machines as well.<br />
<br />
560<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Example 15-13 Output of mpstat command on uni-processor machine (xSeries 342)<br />
<br />
x342rsa:~ # mpstat 2 3 Linux 2.4.21-215-default (x342rsa) 07:12:16 07:12:34 07:12:36 07:12:38 Average:<br />
<br />
CPU all all all all<br />
<br />
%user 1.00 1.00 1.00 1.10<br />
<br />
05/20/04<br />
<br />
%nice %system 0.00 1.50 0.00 1.50 0.00 1.50 0.00 1.55<br />
<br />
%idle 97.50 97.50 97.50 97.35<br />
<br />
intr/s 104.00 104.50 104.00 103.80<br />
<br />
To display three entries of statistics for all processors of a multiprocessor server at one second intervals, use the following command (Example 15-14): mpstat -P ALL 1 3 Example 15-14 Output of mpstat command on four-way machine (xSeries 232)<br />
<br />
[root@x232 root]# mpstat -P ALL 1 10 Linux 2.4.21-9.0.3.EL (x232) 05/20/2004 02:10:49 02:10:50 02:10:51 02:10:52 Average: Average:<br />
<br />
PM PM PM PM<br />
<br />
CPU all all 0 all 0<br />
<br />
%user 0.00 0.00 0.00 0.00 0.00<br />
<br />
%nice %system 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br />
<br />
%idle 100.00 100.00 100.00 100.00 100.00<br />
<br />
intr/s 102.00 102.00 102.00 103.70 103.70<br />
<br />
For the complete syntax of the mpstat command, issue: mpstat -?<br />
<br />
15.14 System x Performance Logger for Linux IBM System x Performance Logger for Linux (xPL, formerly known as PLPerf) is a parameter driven command line tool that collects performance counters from /proc on Linux into a CSV file, which is then readable by Windows Performance Monitor. It allows you to collect Linux performance counters and analyze the data using Windows Performance Monitor (perfmon). xPL is written for x86 and x86_64 based Linux platforms and works on kernels 2.4 and 2.6 independent of the Linux distribution. Refer to 14.1, “Performance console” on page 472 if you are not familiar with perfmon.<br />
<br />
Chapter 15. Linux tools<br />
<br />
561<br />
<br />
You can download the xPL binary and sources from: http://www.pc.ibm.com/support?page=MIGR-64369 xPL performs the following tasks, providing flexible and powerful monitoring: <br />
<br />
Trace counter data for specified time intervals or log file sizes Trace intervals down to milliseconds Allows breaking down the log into multiple files Allows overwriting the same log file (semi-circular) Allows creation of new parameter files Allows saving system information in a separate file<br />
<br />
The following counters can be monitored. Each has multiple variables: <br />
<br />
CPU Interrupts Disk Memory Network<br />
<br />
15.14.1 Counters descriptions Because Windows does not interpret the entered values the same way that Linux does, it is recommended that you do not use the Windows counters description to analyze the data. This section lists the counters that are collected by xPL and their definition, seen by a Linux system. Some of them are specific to the 2.6 kernel.<br />
<br />
CpuStat General CPU counters, which includes the following:<br />
<br />
%User Time Per processor and total of all. Percent of total CPU time spent executing user processes.<br />
<br />
% System Time Per processor and total of all. Percent of total CPU time spent executing system (kernel) processors. Note: In 2.6 kernel, the %System Time that is reported by xPL includes IRQ and SoftIRQ times.<br />
<br />
% Idle Time Per processor and total of all. Percent of total CPU time being idle.<br />
<br />
562<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
% Iowait Time (2.6 kernel only) Per processor and total of all. Percent of total CPU time waiting for I/O to complete. % IRQ Time (2.6 kernel only) Per processor and total of all. Percent of total CPU time servicing interrupts % SoftIRQ Time (2.6 kernel only) Per processor and total of all. Percent of total CPU time servicing softirqs Note: Under Linux, a softirq is an interrupt handler that runs outside of the normal interrupt context, runs with interrupts enabled, and runs concurrently when necessary. The kernel runs up to one copy of a softirq on each processor in a system. Softirqs are designed to provide more scalable interrupt handling in SMP settings.<br />
<br />
% Processor Time Per processor and total of all. Sum of user and system time.<br />
<br />
Context Switches/ sec Total of all CPUs. Total number of context switches across all CPUs. Note: A context switch (also sometimes referred to as a process switch or a task switch) is the switching of the CPU from one process or thread to another. A context is the contents of a CPU's registers and program counter at any point in time. Context switching can be described in slightly more detail as the kernel performing the following activities with regard to processes including threads) on the CPU: Suspending the progression of one process and storing the CPU's state (that is, the context) for that process somewhere in memory. Retrieving the context of the next process from memory and restoring it in the CPU's registers. Returning to the location indicated by the program counter (that is, returning to the line of code at which the process was interrupted) in order to resume the process.<br />
<br />
Chapter 15. Linux tools<br />
<br />
563<br />
<br />
IntStat Provides interrupts counters per second per active interrupt per processor and totals of all per interrupt and per processor. Devices mapped to each interrupt is displayed after the interrupt number in the trace.<br />
<br />
DskStat General Physical disk statistics. Each stat is provided per Physical disk and total of all.<br />
<br />
Reads/sec Total number of read operations completed successfully per second.<br />
<br />
Writes/sec Total number of write operations completed successfully per second.<br />
<br />
Transactions/sec Sum of read and write operations per second.<br />
<br />
Read bytes/sec Total number of bytes read successfully per second.<br />
<br />
Write bytes/sec Total number of bytes written successfully per second.<br />
<br />
Bytes/sec Sum of read and write bytes per second.<br />
<br />
Average bytes/read Total number of bytes read successfully per total # of reads completed successfully. Average Bytes/Write Total number of bytes written successfully per total # of writes completed successfully.<br />
<br />
Average Bytes/Transaction Sum of reads and writes.<br />
<br />
Average sec/read This is the total number of seconds spent by all reads per total # of reads completed successfully.<br />
<br />
Average sec/write This is the total number of seconds spent by all writes per total # of writes completed successfully.<br />
<br />
564<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Average sec/transaction Sum of reads and writes.<br />
<br />
I/O operations in progress Snapshot™ of current disk queue. Incremented as requests are given to appropriate request queue and decremented as they finish. It is the number of requests outstanding on the disk at the time the performance data is collected. It also includes requests in service at the time of the collection. This is a instantaneous snapshot, not an average over the time interval.<br />
<br />
MemStat Provides memory counters.<br />
<br />
Total MB Total physical memory.<br />
<br />
Free MB Total un-allocated physical memory. This is not a good indicator of available physical memory. To check to see if you're out of available physical memory watch for paging or swapping activity.<br />
<br />
Page In KB/sec Total number of kilobytes the system paged in from disk per second.<br />
<br />
Page Out KB/sec Total number of kilobytes the system paged out to disk per second.<br />
<br />
Page KB/sec Sum of Page In and Page Out KB/sec.<br />
<br />
Swap In/sec Total number of swap pages the system brought in per second.<br />
<br />
Swap Out/sec Total number of swap pages the system brought out per second.<br />
<br />
Swaps/sec Sum of Swap In and Swap Out /sec. The fundamental unit of memory under Linux is the page, a non-overlapping region of contiguous memory. All available physical memory is organized into pages near the end of the kernel's boot process, and pages are issued to and revoked from processes by the kernel's memory management algorithms at runtime. Linux uses 4 KB pages for most processors.<br />
<br />
Chapter 15. Linux tools<br />
<br />
565<br />
<br />
Paging and swapping both refer to virtual memory activity in Linux. With paging, when the kernel requires more main memory for an active process, only the least recently used pages of processes are moved to the swap space. The page counters are similar to Memory Page Reads and Writes in Windows. Swapping means to move an entire process out of main memory and to the swap area on hard disk, whereby all pages of that process are moved at the same time.<br />
<br />
NetStat Provides network counters. Each stat is provided per network device and total of all:<br />
<br />
Packets Sent/sec Packets Received/sec Packets/sec (sum of sent and received) Bytes Sent/sec Bytes Received/sec Bytes/sec (sum of sent and received)<br />
<br />
15.14.2 Instructions Before using xPL, you need to set the parameters in the parameter file. the default file is called input.prm, but you can rename it. (If you do rename it, be sure that you refer to the correct file when executing xPL.) See 15.14.2, “Instructions” on page 566 for information about the syntax of the file. To run xPL, you can simply run the xpl command, followed by the parameters file: xpl parameter_file After you run xPL, you can use the created output files with Windows System Monitor. The generated files are in CSV format. Refer to Windows System Monitor (perfmon) help for instructions on how to import a CSV log file. Tip: Start perfmon by clicking Start → Settings → Control Panel → Administrative Tools → Performance, and then click the cylinder icon on the toolbar. You can also generate parameter files by issuing the following command: xpl -g parameter_file You can create a system info file (a file that includes the system configuration information) by issuing the following command: xpl -s system_info_file<br />
<br />
566<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Tip: You can stop xPL by pressing Ctrl+C to perform a graceful shutdown.<br />
<br />
15.14.3 Parameter file The parameter file is the file that is read when xPL is launched. It includes the information about what counters are monitored and how xPL should monitor these counters. The file is divided in several sections: config section Tracing configuration. Options are: Time limited trace Log file size limited<br />
<br />
0 1<br />
<br />
Note: If xPL is unable to write any further to disk, (for example, if the disk is full), it will stop. Trace interval consists of interval in seconds and interval in milliseconds. Total of the two is used for trace interval and are separated for convenience. These values are required by all trace configurations. int_s Trace interval in seconds int_m Trace interval in milliseconds, must be 0 or between 20 and 999. Trace duration is only required by config 0 (Time limited trace). dur_s Trace duration in seconds dur_m Trace duration in milliseconds must be 0 or between 20 and 999. log_size Log file size limit in MB, applies to all configs. Set to 0 for no limit (stops when disk full or time limit has reached.) Note: on average xPL writes 3 KB per sample. new_log 0: no; 1: yes Start a new log file when log file size limit is reached, for all configs. If set to no (0) along with config 0, xPL will overwrite the file if log_size limit has reached so long as the time limit is not reached. If set to yes (1) xPL will<br />
<br />
Chapter 15. Linux tools<br />
<br />
567<br />
<br />
create a new log file incrementing the log file number (see log_start), if used along with config 1 xPL will not stop until manually stopped or when disk is full. log_start Starting number of log file, applies only if new_log is set to yes(1). output_file Output file name, xPL will append .csv extension to the end. No spaces here, xPL will only use the first portion before first space. output_buffer Output buffer size in KB. If set to a number other than zero xPL will wait until buffer is full before writing to disk. counter List of counters to trace, 1 to trace and 0 to not trace. The counters are the one listed earlier. Example 15-15 shows an example of the parameter file you can use with xPL. Using this file, you will monitor the CPUs only, during five seconds, every second. The log file is named output.csv and is overwritten each time you launch xPL. Example 15-15 Sample xPL Parameter file<br />
<br />
# Parameter Input File for IBM xSeries Performance Logger for Linux -xPL (v 1.0.1) # # Trace Parameters # 0: Time limited # 1: Log file size limited config 0 # Note you can use the interrupt signal to stop tracing at any time<br />
<br />
# Time Interval, applies to all configs # interval, seconds int_s 1 # interval, milliseconds, 0 or between 20 and 999 int_m 0 # Trace duration, applies to config 0 # trace duration, seconds dur_s 5 # trace duration, milliseconds, 0 or between 20 and 999<br />
<br />
568<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
dur_m<br />
<br />
0<br />
<br />
# Log file size limit in MB, applies to all configs. # Set to 0 for no limit (stops when disk full or time limit has reached.) log_size 0 # Start a new log file when log file size limit has reached, for all configs. # If set to no along with config 0, xPL will overwrite the file # if log_size limit has reached so long as the time limit is not reached. # If set to yes along with config 1, xPL will continue tracing until disk is # full or manually stopped. # 0: no # 1: yes new_log 0 # Starting number of log file # Applies only if new_log is set to 1. log_start 0 # Log file name, no spaces (xPL will append .csv to the end) output_file output # Log file buffer size in KB (0 means write on every sample) output_buffer 8<br />
<br />
# Set of counters to trace # 0: don't trace # 1: trace counter CpuStat 1 counter IntStat 0 counter DskStat 0 counter MemStat 0 counter NetStat 0<br />
<br />
Chapter 15. Linux tools<br />
<br />
569<br />
<br />
Figure 15-10 shows an example of an activity log file that is generated with xPL on a dual-processor server. The log file (CSV file) has been opened with perfmon.<br />
<br />
Figure 15-10 Trace log from xPL into Windows perfmon<br />
<br />
570<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Note: Perfmon can handle partially written samples at the end of the log.You can use the relog command on Windows to manipulate log files (for example, to convert csv logs to binary). Refer to the relog help for more info. Data is generated as it is read from /proc, so there is no limit on how often you can read them, although it might affect your system performance if the data is read too often.<br />
<br />
15.15 The nmon tool The nmon tool is an easy-to-use monitoring tool developed for AIX® platforms and that has been ported onto Linux. This tool provides a large amount of information within a single screen. Even though it is not supported officially by IBM, it is used during benchmarks to analyze bottlenecks or production systems to give a quick view of system utilization. With nmon, you are able to monitor: <br />
<br />
CPU utilization Memory use Kernel statistics and run queue Disks I/O rates, transfers, and read/write ratios File system size and free space Disk adapters Network I/O rates, transfers, and read/write ratios Paging space and paging rates Machine details, CPU, and operating system specification Top processors User defined disk groups<br />
<br />
You can log the activity of a server using nmon. The generated log file is a comma separated values (CSV) file that you can import and analyze through a spreadsheet. There is a Windows Excel® spreadsheet to automate this process. For more information, see “Data collection mode” on page 575. The nmon tool reads the information from the server in /proc file system. The /proc file system is used by the kernel to talk with he different processes. It is a real-time file system and it is basically where all activity counters are. The /proc file system resides in memory, not on the disk, which means that reading this file system is efficient.<br />
<br />
Chapter 15. Linux tools<br />
<br />
571<br />
<br />
Supported operating systems are: <br />
<br />
SUSE Linux Enterprise Server 8 and 9 Debian Fedora Red Hat 9, EL 2.1, 3 and 4, Knoppix Linux on POWER™ Linux on System z™<br />
<br />
You can download nmon from: http://www-941.haw.ibm.com/collaboration/wiki/display/WikiPtype/nmon<br />
<br />
15.15.1 Using nmon Installing nmon is very easy, because it is a binary that is compiled for every supported distribution. The syntax for using nmon is simple, and you need to specify only a few parameters. There are two different ways of using nmon: Interactive mode Data collection mode<br />
<br />
Interactive mode Interactive mode lets you use nmon as a monitoring tool without any logging of data. Interactive mode is useful for real-time diagnostics and to check quickly the impact of changing parameters. To run the monitoring, just run nmon without any parameter.<br />
<br />
572<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 15-11 shows the welcome screen, which provides the keys to use to display the counters. The values that are displayed are refreshed every two seconds by default, but you can change the refresh time if you want.<br />
<br />
Figure 15-11 The nmon welcome screen<br />
<br />
Chapter 15. Linux tools<br />
<br />
573<br />
<br />
As you press the keys corresponding to the counters that you want to monitor, new information appears. For example, if you press c, information for the CPU appears, d for Disks, n for network, and so on. Figure 15-12 is an example of nmon monitoring CPU, disks, memory, and network components.<br />
<br />
Figure 15-12 Monitoring server’s activity with nmon<br />
<br />
574<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Data collection mode The second way that you can use nmon is with trace logs in data collection mode. This mode allows you to run nmon for a specified period of time and to trace the activity within given intervals (in seconds). This mode generates a log file that you can use for later analysis and performance reports. For example, the most simple way to use nmon to monitor all components every 10 seconds for one hour is to issue the following command: nmon -f -s10 -c360 This command appends all counters in a file called <hostname>_YYYYMMDD_HHMM.nmon, every 10 seconds, 360 times. The -f argument stands for file. If you specify -f, then it means you are using nmon in data collection mode and that you should specify, at least, the interval in seconds -s and the number of occurrences or count -c. Note: If you omit the -s and -c arguments when using nmon in data collection mode, nmon will use the default values, which are 300 seconds and 288 occurrences. This corresponds to a 24-hour run. You can use the default file name when tracing activity (in that case, a new file is created each time you launch nmon), or you can specify your own file name (which can be overwritten if it already exists). To do so, use the -F flag (uppercase) instead of -f, followed by your own user-defined file name. Tip: You can use both data collection mode and interactive mode at the same time. Just launch nmon in data collection mode (with -f or -F) and then in interactive mode to log the activity while monitoring it. After the data collection is finished, you can use a very simple program called nmon2csv to translate your nmon log file into a CSV file. The syntax is: nmon2csv <filename.nmon> The nmon2csv binary is available with the nmon tools downloads at: http://www-941.haw.ibm.com/collaboration/wiki/display/WikiPtype/nmon As a result, you will have a new file with the CSV extension. You can keep this file as activity logs, and you can use it with additional tools.<br />
<br />
Chapter 15. Linux tools<br />
<br />
575<br />
<br />
15.15.2 The nmon Analyser Excel macro Also available is the nmon Analyser Excel macro, which you can use to generate graphs in Excel. You can download the nmon Analyser from: http://www-941.ibm.com/collaboration/wiki/display/WikiPtype/nmonanalyser Tip: If the monitored system and the Windows system have different regional settings, you need to specify, among other parameters, what character is used to separate values (period, comma, or semicolon) and what character is used to separate decimals, in the nmon Analyser Excel spreadsheet. When you run the macro, you get a spreadsheet for each component of your system, including one or more graphs that represents the counters. For example, Figure 15-13 shows the CPU activity summary (on all processors or cores) and the disks transfers for a 20-minute period. System Sum m ary System x 13/06/06 CPU%<br />
<br />
IO/sec<br />
<br />
100<br />
<br />
300<br />
<br />
90 250<br />
<br />
80<br />
<br />
200<br />
<br />
60 50<br />
<br />
150<br />
<br />
40 100<br />
<br />
30 20<br />
<br />
50<br />
<br />
10 0<br />
<br />
Figure 15-13 System summary graph generated with nmon Analyser<br />
<br />
576<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
17:16<br />
<br />
17:15<br />
<br />
17:14<br />
<br />
17:13<br />
<br />
17:12<br />
<br />
17:11<br />
<br />
17:10<br />
<br />
17:09<br />
<br />
17:08<br />
<br />
17:07<br />
<br />
17:06<br />
<br />
17:05<br />
<br />
17:04<br />
<br />
17:03<br />
<br />
17:02<br />
<br />
17:01<br />
<br />
17:00<br />
<br />
16:59<br />
<br />
16:58<br />
<br />
16:57<br />
<br />
0<br />
<br />
Di k f<br />
<br />
usr%+sys%<br />
<br />
70<br />
<br />
Although the system summary provides a good overview of your system activity, you have a set of graphs for each component of your server (CPU, disks, memory, and network). More accurate data is then available. Figure 15-14 shows the network I/O on the system. Network I/O PRDWEB01 (KB/s) 13/07/05 lo-read<br />
<br />
eth0-read<br />
<br />
eth1-read<br />
<br />
lo-write<br />
<br />
eth0-write<br />
<br />
eth1-write<br />
<br />
5000 4500 4000 3500 3000 2500 2000 1500 1000 500 17:16<br />
<br />
17:15<br />
<br />
17:15<br />
<br />
17:14<br />
<br />
17:13<br />
<br />
17:13<br />
<br />
17:12<br />
<br />
17:11<br />
<br />
17:11<br />
<br />
17:10<br />
<br />
17:09<br />
<br />
17:09<br />
<br />
17:08<br />
<br />
17:07<br />
<br />
17:07<br />
<br />
17:06<br />
<br />
17:05<br />
<br />
17:05<br />
<br />
17:04<br />
<br />
17:03<br />
<br />
17:03<br />
<br />
17:02<br />
<br />
17:01<br />
<br />
17:01<br />
<br />
17:00<br />
<br />
16:59<br />
<br />
16:59<br />
<br />
16:58<br />
<br />
16:57<br />
<br />
16:57<br />
<br />
0<br />
<br />
Figure 15-14 Network activity sample graph<br />
<br />
The nmon help file includes all commands that you can use in interactive-mode or data-collect-mode. It also includes some very useful hints. The nmon download site includes useful information, the nmon manual and FAQ, hints and tips, as well as an active forum for questions. To find this site, take the Performance Tools link at: http://www.ibm.com/systems/p/community/<br />
<br />
Chapter 15. Linux tools<br />
<br />
577<br />
<br />
578<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
16<br />
<br />
Chapter 16.<br />
<br />
ESX Server tools Virtualization brings many advantages. However, analyzing performance issues on a virtual machine is more complex than on a conventional system because there is one more factor. Without sound measurements, attempts to tune the ESX Server system or any virtual machine remain pure speculation. A virtualized platform such as ESX Server poses a special challenge when it comes to performance monitoring. Most monitoring tools have difficulties in identifying system performance and bottlenecks when installed in the Console OS. Thus, it is important to understand that the Console OS is simply a very privileged virtual machine with special interfaces to the VMware kernel. Issuing typical Linux performance monitoring commands (such as top) will only reveal the virtualized performance of the Console OS. In addition, other monitoring tools will make it difficult to understand the concept of page sharing that is implemented in VMware and can thus produce erroneous values. It is generally sensible, however, to use application benchmarking tools for virtual machines with the limitation that VMware is not designed to deliver peak performance but rather scalable performance over multiple virtual machines.<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
579<br />
<br />
To monitor an ESX Server system, a number of tools come with ESX Server that can help you analyze bottlenecks and facilitate proactive capacity planning. We discuss two such tools in this chapter, esxtop and vmkusage, but our focus is on the esxtop utility. In our experience, this utility is the best tool for monitoring hardware utilization. This chapter includes the following topics: 16.1, “The esxtop utility” on page 580 16.2, “The vmkusage utility for performance monitoring” on page 588 A note about VMware VirtualCenter The tools for performance measurement that we have introduced so far come free with your ESX Server 2.1.0 and later license. If you require highly detailed, automated performance reports and if you are also interested in receiving alerts in case of a bottleneck situation want to react swiftly to such an event, the VMware VirtualCenter offering might be an appropriate addition. VirtualCenter is an additional product offering of VMware that is not shipped with the standard ESX Server license. VirtualCenter offers numerous features beyond performance management and is therefore beyond the scope of this redbook. For more information about VirtualCenter, see: http://www.vmware.com/virtualcenter<br />
<br />
16.1 The esxtop utility The esxtop utility comes as a standard tool with ESX Server 2.1 and later. This utility enhances the classic top with the awareness of a virtualized system. When used in the Console OS, the esxtop utility reveals the current CPU and memory utilization. It also displays the various processes or virtual machines that run in the system and displays their relative impact on the overall system performance. While esxtop delivers only current information, it is very easy to use and can be of help on a system that has a performance bottleneck. The explanations in this redbook are based on esxtop for ESX Server 3.0. Before you start troubleshooting a performance issue, you should ensure that you have a secure shell client (ssh) available. VMware recommends that Windows users use the PuTTY client, which you can download at no charge from: http://www.chiark.greenend.org.uk/%7Esgtatham/putty/download.html<br />
<br />
580<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
In this chapter, we use PuTTY and ESX Server 3.0. If you are using other versions, their might be some differences.<br />
<br />
16.1.1 Starting esxtop To start the esxtop utility, do the following: 1. Open your SSH client such as PuTTY. 2. Enter the IP Address of the ESX Server system that you want to monitor and select the SSH protocol (Figure 16-1). The standard port for SSH is 22. If your client does not add this port automatically, you will need to type it in.<br />
<br />
Figure 16-1 PuTTY configuration<br />
<br />
3. Open the connection. If you are using the standard security settings, you are not allowed to log on through SSH as root user. So, you need to log on as a different user than root. If you want to log on as root, you have to edit the /etc/ssh/sshd_config file before you log on to allow root access. However, we recommend that you use a regular account.<br />
<br />
Chapter 16. ESX Server tools<br />
<br />
581<br />
<br />
4. Switch to the root user. If you do not, you will not be able to run esxtop as shown in the error screen in Figure 16-2. login as: Daniel Daniel@10.0.0.1’s password: [Daniel@esx Daniel]$ esxtop esxtop: Need to run as user root [Daniel@esx Daniel]$ su Password: [root@esx Daniel]# Figure 16-2 The SSH login<br />
<br />
5. Start the utility by entering esxtop. The esxtop screen opens, as shown in Figure 16-3. The screen shows all processes that are running currently on the ESX Server.<br />
<br />
582<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 16-3 The esxtop start screen<br />
<br />
Chapter 16. ESX Server tools<br />
<br />
583<br />
<br />
To get a list of all the available commands, enter h on the command line to get a screen as shown in Figure 16-4. Esxtop version 3.0.0 Secure mode Off Esxtop: top for ESX These single-character commands are available: ^L space h or ? q<br />
<br />
-<br />
<br />
redraw the screen update display help; show this text quit<br />
<br />
Interactive commands are: fF oO s # W e<br />
<br />
Add or remove fields Change the order of displayed fields Set the delay in seconds between updates Set the number of instances to display Write configuration file ~/.esxtop3rc Expand/Rollup Cpu Statistics<br />
<br />
Sort by: U:%USED Switch display m:ESX memory<br />
<br />
R:%RDY<br />
<br />
N:Default<br />
<br />
d:ESX disk<br />
<br />
n:ESX nic<br />
<br />
Figure 16-4 esxtop help<br />
<br />
The default is to display CPU usage, If you prefer to capture the memory, disk, or NIC usage, the last line of the display (Figure 16-4) shows the keys to press to switch: Press m to display memory usage Press d to display disk usage Press n to display network usage In this section, we explain how to use esxtop with the CPU usage display. However, if you want to use another option, you can switch, and the operation is the same.<br />
<br />
584<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
By entering the f command on the start screen, the screen to reorder the fields opens, as shown in Figure 16-5. To reorder the fields, enter the corresponding letter. To exit and go back to the normal esxtop screen, enter any key other than A through H. Current Field order: ABCDEfgH * * * * *<br />
<br />
A: B: C: D: E: F: G: * H:<br />
<br />
ID = Id GID = Group Id NAME =Name NMEM = Num Members %STATE TIMES = CPU State Times EVENT COUNTS/s = CPU Event Counts CPU ALLOC = CPU Allocations SUMMARY STATS = CPU Summaryh Stats<br />
<br />
Toggle fields with a-h, any other key to return: Figure 16-5 Field order screen<br />
<br />
The esxtop utility also offers the ability to run in batch mode. For example, use this command: esxtop -b -n iterations > logfile For information about how to use this command, enter the following on the SSH command line: man esxtop<br />
<br />
16.1.2 Using esxtop with ESX Server 3.0 After you have customized the esxtop main screen, you can start analyzing the bottlenecks in your system. Note: Remember that the examples that we use here are based on the CPU usage, but you can also monitor memory, disk, and network usage with this tool.<br />
<br />
Chapter 16. ESX Server tools<br />
<br />
585<br />
<br />
If you look at the CPU usage main screen, Figure 16-6, the first line (highlighted in a red box) shows you the load average for all physical CPUs on the ESX Server machine. A load average of 1.0 means that the physical CPU is fully utilized. So, if this value is under 1.0, the CPU is under utilized. If this value is over 1.0, the CPU is over utilized, and you need to increase the number of physical CPUs or to decrease the number of virtual machines that are running on the server.<br />
<br />
Figure 16-6 CPU load average<br />
<br />
The second line of the screen shows the PCPU usage—the usage for each individually installed CPU. The last value is the average percentage for all the CPUs. If you are using a multi-core CPU, each core is displayed as a separate CPU. If this value is under 80% utilization, your system should perform well. If this value is about 90% or higher, this is a critical warning that the CPU is overloaded. Figure 16-7 shows a system that has a CPU bottleneck.<br />
<br />
Figure 16-7 PCPU usage<br />
<br />
586<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
The NAME column provides the given name of your virtual servers, and each line provides information about the CPU usage. For our discussion, we concentrate only on the %USED and %RDY fields. Depending on your view, you might have to expand your SSH client window to see the %RDY field. The explanation of these fields are as follows: %USED This field shows the percentage of physical CPU resources that are used by a virtual CPUs. If the virtual CPU is running at the full capacity of the physical CPU, you can identify the virtual machine that might be causing the bottleneck. %RDY This field gives you information about the time that a virtual machine was ready but could not get scheduled to run on a physical CPU. As a rule of thumb, this value should remain under 5%. If you are running into a CPU bottleneck on a virtual machine, the most common solutions are: Increasing the number of CPUs or cores Decreasing the number of virtual machines Moving the virtual machine to another ESX Server system<br />
<br />
Memory usage As described in 16.1.1, “Starting esxtop” on page 581, you can change the esxtop view to show the memory usage by pressing m. We also recommend that you monitor the following values: The maximum available memory that is used by the virtual machine The amount of swapped memory that is used by a virtual machine. If the machine swaps some data out that is not bad because that data might be inactive pages and the swap percentage increases, it could be a sign of a performance problem.<br />
<br />
16.1.3 Exiting esxtop To exit the utility, enter q.<br />
<br />
Chapter 16. ESX Server tools<br />
<br />
587<br />
<br />
16.2 The vmkusage utility for performance monitoring Another monitoring tool is the vmkusage utility. This utility displays actual and historical performance data that is not as detailed as the information that esxtop displays but the information is presented in a graphical manner. The vmkusage utility comes standard with ESX Server 2.1.0 to 2.5x versions. ESX Server 3 or later obtains statistics through VirtualCenter, and there is no more support for the vmkusage utility. The vmkusage utility is installed by default, but you first need to activate it with the following command: vmkusagectl install With this command, the vmkusage utility creates performance reports in the /var/log/vmkusage/ directory and continues to update those reports every five minutes with the most current values. You can then view the data graphically. You can access the graphical performance data through a Web browser to the address of your ESX Server server. For example: http://vmware.itso.ral.ibm.com/vmkusage/ Figure 16-8 shows an example of the graphical output from vmkusage. The values in this figure show the actual and long-term utilization of the ESX Server system and all running virtual machines. The various graphs that are associated with each subsystem give a more complete picture of the behavior of both the system and the running virtual machines. Restriction: The vmkusage utility displays performance data only for virtual machines that are currently running.<br />
<br />
588<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 16-8 The graphical presentation of the vmkusage tool<br />
<br />
In the VM Summary table (the top of Figure 16-8), you see all currently running virtual machines. Here you get an overview which resources are dedicated to each machine. In the graphical view (the lower half of Figure 16-8), various graphs are showing the use of each machine. The first line is the physical machine, and below there is one line for each virtual machine. So you get an overview that is the utilization of the physical machine and how the utilization is distributed to the virtual machines, or which machine can cause a performance bottleneck. If you analyze this screen in regular intervals and compare it with the values from the last time you are able to determine performance trends and whether there is a chance you’ll run into a bottleneck.<br />
<br />
Chapter 16. ESX Server tools<br />
<br />
589<br />
<br />
590<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
17<br />
<br />
Chapter 17.<br />
<br />
Capacity Manager Capacity Manager is an efficient system management tool which is part of IBM Director. Capacity Manager can help you to measure the potential bottlenecks of multiple IBM Director systems. You can use this tool to forecast performance degradation of a server and its subsystems. You can plan for an appropriate action to overcome the bottleneck well in advance, so as to prevent overall performance degradation. Capacity Manager is supported on all IBM Director Agents running under Linux, NetWare, or Windows operating systems. For a complete reference of all supported operating systems, review the Installation and Configuration Guide that is located in the /docs directory on the IBM Director CD-ROM. This chapter includes the following sections: <br />
<br />
17.1, “Introduction” on page 592 17.2, “Capacity Manager data files” on page 593 17.3, “Installing Capacity Manager” on page 594 17.4, “Monitor Activator” on page 596 17.5, “Report Generator” on page 597 17.6, “Report Viewer” on page 607 17.7, “Performance analysis” on page 617<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
591<br />
<br />
17.1 Introduction From the moment you install Capacity Manager on a managed system (running IBM Director Agent), performance data is collected in the background. Over time, key resource utilizations are collected from network systems and merged into a single report that you can view graphically or that you can export for further analysis. With IBM Director, Capacity Manager also supports scheduled exporting. This feature enables you to export performance reports automatically on a regular basis, for example to an intranet server where the data is accessible for multiple users with a simple Web browser. These reports show potential capacity bottlenecks within the selected systems. Your analysis and the ability to predict bottlenecks is critical when planning for future upgrades. Capacity Manager provides the ability to plan the allocation of hardware upgrades for the systems that really need upgrading before a capacity bottleneck occurs. Using Capacity Manager puts you in the position to manage your systems proactively rather than reactively. Key concept: The key concept to understanding Capacity Manager is that the data is always being gathered. Thus, you do not have to start the logging of data. With Capacity Manager, you simply specify what data you want retrieved from the servers and workstations in your network, and that data is gathered up and displayed graphically for you. Up to one month’s worth of data is saved automatically by every system. Capacity Manager is available as a fee-based extension for IBM Director. For more information about using IBM Director, review the Implementing IBM Director 5.10, SG24-6188. You can find more information about Capacity Manager at: http://www.ibm.com/systems/management/director/extensions/capm.html Table 17-1 lists the part numbers for Capacity Manager. Table 17-1 Capacity Manager part numbers<br />
<br />
592<br />
<br />
Description<br />
<br />
Part number<br />
<br />
Capacity Manager 5.10 Agent + 1 year Software Subscription<br />
<br />
32R1225<br />
<br />
Capacity Manager 5.10 Media Pack<br />
<br />
32R1230<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
17.2 Capacity Manager data files Capacity Manager uses two types of data files: Raw data files, with the extension SLT Report files, with the extensions CMR, TXT, HTML, XML and GIF When you use the Report Generator, it uses SLT files from the various systems that you specify and the report definition. Then it builds a report file (a .CMR). Figure 17-1 illustrates this process. Stored on every IBM Director system<br />
<br />
REALTIME.SLT Last 24 hours at 1 minute intervals TREND.SLT Last month at 5 minute intervals<br />
<br />
Inputs to report generation<br />
<br />
Report definitions<br />
<br />
CMR report file<br />
<br />
Selected systems<br />
<br />
Figure 17-1 Generating report files<br />
<br />
Capacity Manager also uses CMUserSettings.properties, which is created when the user makes changes to the default settings. SLT files Capacity Manager saves one calendar month of data automatically. The data is stored in two .SLT (or slot) files. These files are stored in the SLTFILES directory. – REALTIME.SLT This file includes data from the last 24 hours, stored at one-minute intervals. Data that is older than 24 hours is discarded. The data is actually retrieved from the system information service of Netfinity Manager at one-minute intervals. – TREND.SLT This file includes data from the last calendar month, stored at five-minute intervals. For example, on October 15, the TREND.SLT file includes data dating back to September 15. Data older than one calendar month is discarded. The values that are stored in this SLT file are the average of the five one-minute values of the last five minutes.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
593<br />
<br />
On all machines with Netfinity Manager installed, the SLT files are continually being updated with the latest data. CMR files CMR files are the output of the report generation process. They include the data specified by the selected MON file and for the systems chosen during the process.<br />
<br />
17.3 Installing Capacity Manager Capacity Manager Agent is an fee-based extension for IBM Director. Capacity Manager can be installed manually as a Windows MSI file, Linux RPM file, or as a NetWare packet. You can also install Capacity Manager using scripting or the Software Distribution mechanism of IBM Director. You also need to install the support for Capacity Manager on your management server and the IBM Director Console. After a successful installation, your console should look similar to that shown in Figure 17-2 on page 595. To collect data on an IBM Director Agent, you must have the following components installed on that system (installed in this order): IBM Director Agent Capacity Manager Agent<br />
<br />
594<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 17-2 IBM Director management console: Capacity Manager tasks<br />
<br />
You can perform four functions from this menu: 1. Double-click any of the four Using Capacity Manager tasks (as shown in Figure 17-2) to learn about Capacity Manager: – – – –<br />
<br />
New features Overview Report Viewer tour Steps to create a report<br />
<br />
If you are new to IBM Director and Capacity Manager, we suggest that you review each of these help topics. 2. Change what data is recorded on specific clients using Monitor Activator (see 17.4, “Monitor Activator” on page 596). 3. Generate a report either directly to the viewer or to a report file using Report Generator (see 17.5, “Report Generator” on page 597). 4. View a report that has already been generated using Report Viewer (see 17.6, “Report Viewer” on page 607).<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
595<br />
<br />
17.4 Monitor Activator The Monitor Activator function is where you specify what data is to be gathered on specific clients or groups of clients. Simply drag the Monitor Activator icon onto a group or a single client to activate it. The window shown in Figure 17-3 opens.<br />
<br />
Figure 17-3 Monitor Activator<br />
<br />
With the Capacity Manager, the following statistics are gathered automatically. By default, the following counters are enabled on Windows systems: <br />
<br />
CPU Utilization and Process count Disk workload and usage Memory usage Network Interface: Bytes Total/sec Network Interface: Packets/sec Physical Disk: %Disk Time<br />
<br />
On Linux Systems the following monitors are enabled by default: <br />
<br />
596<br />
<br />
CPU Utilization Memory usage Network Interface: Bytes Total/sec Network Interface: Packets/sec<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
These monitors are the minimum required to provide performance analysis data as described in 17.7, “Performance analysis” on page 617. You can enable as many monitors as you want; however, notice that enabling too many counters can impact system performance negatively. To enable a monitor: 1. 2. 3. 4.<br />
<br />
Select the monitor (expand monitor groups as required). Click On. Repeat steps 1 and 2 for any additional monitors that you want to enable. Click Apply.<br />
<br />
Note that the process and Performance console counters are specific to Windows systems. They are not available on Linux or NetWare systems.<br />
<br />
17.5 Report Generator With this function, you gather data from specific systems and either display it on-screen using the Report Viewer, or save it to a report file. There are four predefined report definitions, as shown in Figure 17-2 on page 595. These are: <br />
<br />
Daily Report (to viewer) Hourly Report (to viewer) Monthly Report (to file) Weekly Report (to file)<br />
<br />
They can be used as-is, modified by double-clicking them, or deleted. We discuss these in 17.5.2, “Working with predefined reports” on page 604. You can also create new report definitions, as we describe next. After the report definition is ready for use, you actually generate a report by dragging the report definition icon onto a system or a group of systems, which we discuss further in 17.5.3, “Generating a report” on page 605.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
597<br />
<br />
17.5.1 Creating a new report definition To create a new report definition, double-click [New Report Definition] in the IBM Director management console. The window shown in Figure 17-4 opens.<br />
<br />
Figure 17-4 New report definition<br />
<br />
The components of this window are as follows: Report Duration Duration specifies how far back you want your report to measure. Because Capacity Manager keeps one month of data (a calendar month, for example, November 10 to October 10), you can schedule a report to measure up to one month of time. Available choices are: – – – – –<br />
<br />
1 hour 8 hours 1 day 1 week 1 month<br />
<br />
Time periods of one hour look back one hour from the beginning of the current hour but also include whatever time has passed in the current hour.<br />
<br />
598<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
For example, if you run your report at 3:18 p.m., the report measures from 2 p.m. to 3:18 p.m. Time periods of eight hours look back eight hours from the beginning of the current hour but also include whatever time has passed in the current hour. Time periods of one day look back 24 hours from the beginning of the current hour. Time periods of one week or one month look back from the previous midnight. Global Sampling Frequency This determines how often data is collected, and therefore, the granularity of your report data. You can select: – – – – –<br />
<br />
1 minute 5 minutes 30 minutes 1 hour 1 day<br />
<br />
You might want to choose a larger value if you are concerned about the space taken to store this data. Not all of these choices will be available. If you choose a duration of one week or one month, the one minute sampling frequency will not be available. This is because the data for the last 24 hours is saved per minute, but after a day, data is averaged to one value per five minutes. You can also specify a sampling frequency for specific monitors to override this global value by clicking Monitor Selection. Note: Raw data is gathered at one-minute intervals. These one-minute values are instantaneous values and are not averages of the last minute of activity, which means that any spikes in usage that do not continue over a sampling point is not recorded. Collecting Minimum and Maximum Values Collecting minimum and maximum values gathers the highest and lowest value for each monitor within the sampling frequency. Doing so triples the size of report files and slows performance in the Report Viewer, but it provides valuable data, especially if the sampling frequency is set to a large amount of time. Minimum and maximum values are not available when the sampling frequency is set to the smallest sampling frequency available for that time period.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
599<br />
<br />
Note: If you want to display minimum and maximum values in the Report Viewer (see “Showing Minimum and Maximum Values” on page 615), we strongly recommend you collect the minimum and maximum data at this point. If you do not collect the minimum and maximum data but choose to display the minimum and maximum values anyway, then the graphs displayed are approximations based on incomplete data and are likely to be inaccurate. Days and Times The next step is to define when data is collected. To make your report accurately reflect the use of your systems, you can select the days and times they are typically used. To define days, click the check boxes of the days you want included. To define times to measure, click New and enter start and stop times in the window shown in Figure 17-5. To measure the entire day, select 0:00 for both choices. To exclude times when systems are typically not used (for example, the lunch hour) you can define more than one time in this field. Click the Edit button and enter 08:00 for the start time and 12:00 for the end time. Click the New button and enter 13:00 for the start time and 17:00 for the end time.<br />
<br />
Figure 17-5 Setting start and stop times<br />
<br />
Method of Generating a Report Here, you specify whether you want the report to be displayed directly on-screen or stored for further processing. If you want to save the Capacity Manager data for further reference you have multiple options such as storing the report in the Capacity Manager format (only viewable by the IBM Director Console) or saving it to a file (TXT, CSV, HTML and XML are supported). Capacity Manager allows you also to store raw data within a SQL database. You can also schedule an exporting task using the scheduler.<br />
<br />
600<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
With the Generate to file option, later, when you request a report using this definition, the window shown in Figure 17-6 opens, asking whether you want to run the report now or schedule it for later. We discuss scheduling in 17.5.3, “Generating a report” on page 605.<br />
<br />
Figure 17-6 Message shown if the report is to be saved to a file<br />
<br />
There are four file types you can choose from: – CMR: Capacity Manager report format, for use only in the Report Viewer. – TXT: Capacity Manager text format, for use in the viewer or in a spreadsheet application. – HTML: For exporting to a Web server. – XML: For further processing with other tools. If you choose to save the report to a file, the report will be saved in the reports subdirectory on the server. They are not saved on the console system. If you want the report saved on the console, you need to view the report using the Viewer, then use the Save Locally function to save the report to the console’s hard disk. Note: You can change whether the report is saved to disk or viewed on-screen at a later point by simply right-clicking the report definition in the IBM Director management console, as shown in Figure 17-7.<br />
<br />
Right-click the report definition to specify whether the report is to display in the Report Viewer or should be saved to a file.<br />
<br />
Figure 17-7 Changing the report’s output<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
601<br />
<br />
Append time stamp to file name This parameter is set by default. The name of the report file is based on the report definition file. To make the name unique over repeated uses of the same report definition, a time stamp is appended to the report definition to create a unique file name. Monitor Selection Here, you specify which monitors you want to gather: – Choose Include all activated monitors if you want all enabled monitors to be gathered from the clients (as was specified for the client using Monitor Activator, as described in 17.4, “Monitor Activator” on page 596). – Choose Select individual monitors if you want to select a subset of the enabled monitors on each client. Clicking Monitor Selection produces a window similar to that shown in Figure 17-8.<br />
<br />
Figure 17-8 Monitor selection<br />
<br />
602<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Here, you specify which of the enabled monitors you want to include in this report. By default, none is included. To include a monitor, highlight it and click Include.<br />
<br />
Figure 17-9 CPU Utilization monitor now included in the report<br />
<br />
Optionally, you can now override the global sampling frequency you set in the Report Definition window (see Figure 17-4 on page 598). There are two ways to define sampling frequencies: a global sampling frequency, which applies to all monitors, and an individual sampling frequency, which can be set for a particular monitor. You might want to set the frequency of monitors that do not change dramatically during the day to a larger unit of time, such as one day, while setting other monitors that do change dramatically, such as CPU utilization, to a smaller unit of time, such as five minutes. To set an individual frequency, select Override global settings, then specify the individual frequency in the Sampling frequency drop-down list. You can also override the global setting for collecting minimum and maximum values for the specific monitor. Do this by selecting the Collect Min and Max values check box. The default of this check box is the global setting you set in the Report Definition window (see Figure 17-4 on page 598).<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
603<br />
<br />
Timeout Parameter This value (between 1 minute and 9 minutes) is how long Capacity Manager will wait for each client to respond when a report is requested. Now that all parameters are set, click Save As to save the report definition. Specify the name of the report definition as prompted and click OK.<br />
<br />
17.5.2 Working with predefined reports As shown in Figure 17-2 on page 595, four reports are predefined: <br />
<br />
Hourly Daily Weekly Monthly<br />
<br />
These reports gather a predefined subset of all available monitors. Table 17-2 shows the characteristics of each of these reports. Table 17-2 Characteristics of the predefined definition reports<br />
<br />
Hourly<br />
<br />
Daily<br />
<br />
Weekly<br />
<br />
Monthly<br />
<br />
Output<br />
<br />
Report Viewer<br />
<br />
Report Viewer<br />
<br />
File<br />
<br />
File<br />
<br />
Duration<br />
<br />
1 hour<br />
<br />
1 day<br />
<br />
7 days<br />
<br />
30 days<br />
<br />
Sampling Frequency<br />
<br />
Every minute<br />
<br />
Every minute<br />
<br />
Every 5 minutes<br />
<br />
Every 5 minutes<br />
<br />
Hours<br />
<br />
N/A<br />
<br />
24 hours<br />
<br />
0800-1700<br />
<br />
0800-1700<br />
<br />
Monitors<br />
<br />
All enabled monitors<br />
<br />
All enabled monitors<br />
<br />
All enabled monitors<br />
<br />
All enabled monitors<br />
<br />
A duration of one hour starts on the hour but also includes whatever time has passed in the current hour. For example, if you schedule your report at 3:18 p.m., the report measures from 2 p.m. to 3:18 p.m. Durations of eight hours look back eight hours from the beginning of the current hour but also include whatever time has passed in the current hour. Durations of one day look back 24 hours from the beginning of the current hour. Durations of one week or one month look back from the previous midnight. You can do the following with predefined reports: Execute it, just as you can with report definitions you have created, by dragging and dropping one onto a client or group of clients. Edit it by double-clicking the entry in the management console.<br />
<br />
604<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Delete it by right-clicking the entry and clicking Delete. Changing the output definition from viewer to file or from file to viewer by right-clicking it and clicking To viewer or file.<br />
<br />
17.5.3 Generating a report To generate a report, simply drag the report definition onto a client or a group of clients, as shown in Figure 17-10.<br />
<br />
Generate a report by dragging the report definition onto either a specific client or to a group of clients.<br />
<br />
Figure 17-10 Generating a report<br />
<br />
If you choose to output the report to the Report Viewer, you see a progress window showing the status of data being gathered from each client. The Report Viewer then loads and displays the results. We discuss the Report Viewer in 17.6, “Report Viewer” on page 607. If your report definition specified to output to a file, then you see the following dialog box asking if you want to execute it immediately or schedule the task to be executed at a later time.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
605<br />
<br />
Figure 17-11 Output to file dialog box<br />
<br />
Regardless of which you select, the report is saved to a file on the server (not the console). The name of the report is the name of the report definition, plus the time and date the report was created if you selected the option Append time stamp to file name in Figure 17-4 on page 598. Clicking Schedule opens a window where you can specify when you want the task to be run (see Figure 17-12).<br />
<br />
Figure 17-12 Specifying a schedule<br />
<br />
Specify the name that you want to assign to this job. This name is displayed as the job name on the Jobs page in the Schedule window (access this by clicking Scheduler on the management console).<br />
<br />
606<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
By clicking Advanced, you can specify additional requirements for the schedule, including: Whether to repeat the task What to do on systems that are not available at the time the task runs Whether you want an event alert generated in various situations<br />
<br />
17.6 Report Viewer The Report Viewer is used to examine reports that you have requested to be gathered immediately or to examine reports you have saved to a file. The viewer starts automatically if the report definition you used specifies that the output should go to the viewer. To view a report that was saved to a file, double-click the Report Viewer icon from the IBM Director management console (see Figure 17-2 on page 595). You are then prompted to select a report file (.CMR or .TXT) from the IBM Director server’s REPORTS directory. A typical Report Viewer window is shown in Figure 17-13 on page 608. As you can see, it is made up of three window panes.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
607<br />
<br />
System pane (Details view shown)<br />
<br />
Monitor pane Figure 17-13 The three panes of the Report Viewer main window<br />
<br />
608<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Graph pane<br />
<br />
You can adjust the space each pane takes up on the window by dragging the border between two panes with the mouse. The button bar includes the elements shown in Figure 17-14.<br />
<br />
File menu<br />
<br />
Edit menu<br />
<br />
Table view<br />
<br />
Icon view<br />
<br />
Performance analysis (see p. 617)<br />
<br />
Hypergraph view<br />
<br />
Sort the view (field, ascending/descending)<br />
<br />
Report Information<br />
<br />
Help<br />
<br />
Figure 17-14 Report Viewer toolbar<br />
<br />
17.6.1 Setting thresholds You might find it useful to set thresholds for particular monitors. Capacity Manager lets you set a warning threshold, which it displays in yellow, and a critical threshold, which it displays in red. The red and yellow markers appear in the System pane in both the Hypergraph and Details views, as described in 17.6.2, “The System pane” on page 611, and in the Graph pane.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
609<br />
<br />
To set the thresholds, click → Settings then select the Monitors tab. The window shown in Figure 17-15 opens.<br />
<br />
n Select the monitor<br />
<br />
o Select the Critical threshold limit<br />
<br />
p Select the Warning threshold limit<br />
<br />
Help text<br />
<br />
Figure 17-15 Selecting threshold limits<br />
<br />
When you set the thresholds, you see markers in both the System pane and the Graph pane, as shown in Figure 17-13 on page 608. Tip: Do not modify thresholds for bolded monitors. As described in the Help text at the bottom of the window shown in Figure 17-15, the threshold values for key monitors (these are in bold font in the monitor list) are already set to optimum values. We recommend that you do not modify the threshold for these key monitors.<br />
<br />
610<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
17.6.2 The System pane The System pane underneath the toolbar shows the systems that you have chosen in your report. There are four ways of viewing the systems in the system pane: <br />
<br />
Table view (the default) Icon view Hypergraph view Performance analysis<br />
<br />
These choices are available from the toolbar icons, as shown in Figure 17-14 on page 609. You can select one or more systems in the System pane. Doing so assigns a colored circle, triangle, or square to each system that acts as the legend for the display in the Graph pane. The legend allows you to distinguish between systems when you have multiple systems selected. You can select more than one system using either the Shift or Ctrl key.<br />
<br />
Table view The Table view, shown in Figure 17-13 on page 608, lists the average values for all of the monitors you have selected plus system information parameters such as bus type and processor speed. The monitors are also repeated in the Monitor pane. If you click one of the monitor values for a particular system, the Graph pane displays that monitor for that system automatically. You notice in Figure 17-13 on page 608 that there are dashes instead of values for monitors of some systems because that particular monitor is not relevant or not available for that particular system. You might also see a question mark against some monitors for some systems. If, for example, a system has just been installed and has not collected enough data points for the requested period, then you would see a instead of the average value in the view. You might also get a question mark if the SLT file is corrupted or if the IBM Director agent and IBM Director server have different data settings. There are also a number of adjustments that can be made to the way the information is displayed in the Details view: Sorting by column You can sort the systems by any of the columns in the Details view by choosing from the Sort By drop-down menu. You can also click the<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
611<br />
<br />
Ascending or Descending buttons sort order to adjust the way the systems are displayed. Changing the size of the legend icon By default, the Table view shows small icons. You can set large icons by clicking Edit → Settings → Window and selecting Use large icons for systems. Shortening the column titles By default, the full monitor name or system parameter name is displayed at the top of each column in the Details view, which means that you have to scroll horizontally to see all the monitor values. You can specify that only an abbreviation of the column heading be used by clicking Edit → Settings → Window, selecting Abbreviate column headings and then specifying the number of abbreviated characters.<br />
<br />
Icon view Clicking Icon view on the toolbar converts the System pane into a view just showing the names of the systems, such as that shown in Figure 17-16. This view is useful when you have many systems to display and when you are interested only in the Graph pane.<br />
<br />
Figure 17-16 Icon view (large icons)<br />
<br />
Hypergraph view The Hypergraph view displays average values of the selected monitor for all the systems in the report. If you click Descending, those systems with the highest average value are at the top of the report. If you click Ascending, those systems with the lowest average value are at the top of the report.<br />
<br />
Figure 17-17 Hypergraph view<br />
<br />
612<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
The tops of the icons mark the values that are displayed. If you have defined thresholds, they appear as horizontal lines in the Hypergraph view. Tip: You can change the height of the System pane by dragging down the border between it and the other two panes.<br />
<br />
Performance analysis This new function lets you analyze your system for bottlenecks and offers possible ways to improve performance. See 17.7, “Performance analysis” on page 617 for more details.<br />
<br />
17.6.3 The Monitor pane The Monitor pane on the lower left-hand side of the Report Viewer window (see Figure 17-13 on page 608) lists the monitors that you have chosen in the Report Generator. All the monitors that apply to any of the systems that you selected are displayed in the window. You can select only one monitor at a time. The monitor that you select is displayed in graphical format in the Graph pane for the systems that you have selected in the System pane.<br />
<br />
17.6.4 The Graph pane On the lower right-hand side of the Report Viewer window is the Graph pane (see Figure 17-13 on page 608). To make the graph larger, select the edge of the pane with your mouse and drag the panel up. To display data on the graph, select a monitor from the Monitor pane, then one or more systems from the System pane (select more than one system using the Shift or Ctrl key). Figure 17-18 on page 614 shows the CPU Utilization monitor selected.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
613<br />
<br />
Zoom You can zoom in on particular time periods of the graph by clicking the button to activate zoom, then clicking inside the graph that you want to see more closely. Flyover help tells you when zoom is available. To zoom out, right-click inside the graph. If you do not zoom in, then the data that is displayed at each time period is the average of the values for that period.<br />
<br />
To zoom in, click the button to activate zoom, then click in a grid area in the graph to zoom in on that time period.<br />
<br />
Forecast is available.<br />
<br />
Zoom is available.<br />
<br />
Figure 17-18 CPU utilization of multiple systems<br />
<br />
If you have more than one system selected, the graph shows three lines of different colors. The connector (a circle in Figure 17-18) in the graph matches that in the System pane. To change the time scale, select a new value in the Point per drop-down list box in the lower left portion of the graph. This will show more data in the window but can make it too cluttered if you have many systems displayed. You can show a legend box showing the names of each of the lines in the graph. To do so, click Edit → Settings → Graph, then select Show the legend.<br />
<br />
Forecast The Forecast function allows you to see Capacity Manager's prediction for the performance of your selected systems. See 17.7.4, “Forecast” on page 622 for more information.<br />
<br />
614<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Showing Minimum and Maximum Values As stated in “Zoom” on page 614, if you do not zoom in, then the data that is displayed at each time period is the average of the values for that period. When you have only one system selected, you can also display the minimum and maximum values in this situation by clicking Edit → Settings → Graph then clicking Show minimum and maximum lines when averaging. This will display a red line for the maximum value within that time period and a green line for the minimum value within that time period (Figure 17-19). Important: If you want to display minimum and maximum values in the Report Viewer, we strongly recommend that you first turn on the collection of min/max data in the report definition file (see 17.5, “Report Generator” on page 597). If you do not collect the min/max data but choose to display the min/max values anyway, then the graphs displayed are approximations based on incomplete data and are likely to be inaccurate.<br />
<br />
Maximum value in each time period<br />
<br />
Average value for each time period<br />
<br />
Minimum value in each time period<br />
<br />
Figure 17-19 Minimum and maximum values<br />
<br />
Note: Minimum and maximum values do not appear if you have more than one system selected, nor do they appear when you are at the maximum zoom level.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
615<br />
<br />
Trend graph For small numbers of systems, it is appropriate to show a line on the graph for each system. However, with large systems, this can become unmanageable. To compensate for this, Capacity Manager can be configured to group all systems into one graph line and show minimum and maximum values for that time period for all systems, as shown in Figure 17-20.<br />
<br />
Data from individual systems<br />
<br />
Average for all systems Line length shows the range of systems for that data point<br />
<br />
Figure 17-20 A sample trend graph<br />
<br />
The trend graphs plot the average value of the selected monitor for all of the systems you have chosen. For each time period, there is a vertical line: Data from individual systems is represented as dashes. The length of the vertical line represents the range of all the selected systems’ utilization data points. Clusters of points on the line represent a concentration of data. Capacity Manager switches a graph to a trend graph automatically when the number of systems selected exceeds a specified number. That number is set by clicking Edit → Settings → Graph and changing the field Maximum systems to graph individually. The default is 3. Capacity Manager can graph up to nine systems on the chart at the same time. Any number above nine is trended automatically.<br />
<br />
616<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
17.7 Performance analysis Performance analysis is a new artificial intelligence feature that probes for bottlenecks in server hardware performance, diagnoses the problem, and suggests ways to improve performance. The performance analysis algorithm is based on the experiences of experts. The algorithm can find many but not all system problems. A minimum of a month’s worth of data is needed to make accurate predictions. Note: Performance analysis is available on Windows operating systems, Linux and NetWare. The algorithm monitors four server functions: <br />
<br />
Memory Disk subsystem CPU Network<br />
<br />
As described in 17.4, “Monitor Activator” on page 596, the monitors that are activated by default in IBM Director clients are required to perform this analysis (depending on actual operating system installed: <br />
<br />
CPU Monitors: CPU Utilization Memory Monitors: Memory Usage Disk Monitors: Disk Workload and several Disk usage counters Physical Disk: % Disk Time Network Interface: Bytes total/sec Network Interface: Packets/sec<br />
<br />
17.7.1 Reports produced The report produced by the performance analysis function consists of two main sections: Recommendations: a summary of the actions that are recommended Details: all analysis results A bottleneck that is reported in the details section will appear in the recommendations section if it meets one of the following criteria: It occurred on the last day of the report. It occurred more than 25% of the time, and it occurred more than any other bottleneck for that particular system.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
617<br />
<br />
It appears that it will occur in the future; this prediction is based on performance analysis having enough data for the system to make a reliable forecast. The performance analysis function button appears as one of four icons, as described in Table 17-3. Table 17-3 Performance analysis buttons<br />
<br />
Icon<br />
<br />
Meaning The performance analysis report is ready. There are no bottlenecks listed in the recommendations section, but some latent bottlenecks have been detected. The performance analysis report is still being prepared.<br />
<br />
The performance analysis report could not be prepared because you are missing one or more critical monitors.<br />
<br />
The performance analysis report is ready, and you have system bottlenecks discussed in recommendations.<br />
<br />
To see the results of the performance analysis on your data, click the button that appears on the toolbar (Table 17-3). A window similar to the one shown in Figure 17-21 on page 619 appears. The performance analysis report is available online as an HTML file.<br />
<br />
618<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 17-21 Performance analysis report<br />
<br />
The report presents the bottleneck information first as a summary of the recommendations, then in a more detailed format. It also has links to the supporting graphic data. Keep in mind that bottleneck detection and analysis are complicated. If a monitor seems to be missing in one bottleneck, it might be because it is contributing to another one. The report can also be saved to a disk. An x.HTML file is created, where x is the file name the user specifies when saving. It includes links to the performance analysis view information, the report information, and the Table view information.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
619<br />
<br />
17.7.2 Types of bottlenecks Bottlenecks are detected when one or more monitors exceed a programmed threshold setting for an extended period of time. You can adjust these threshold settings, but the default settings, particularly those that are critical for the integrity of the performance analysis, are best left unchanged. The types of bottlenecks are: Bottlenecks A bottleneck that is currently happening is sometimes called a realized bottleneck or just a bottleneck. A bottleneck occurs on a system when one or more devices are constrained. Latent bottlenecks Often, when you fix one bottleneck, there will be another waiting to happen, but it does not occur because the system was slowed down by the first bottleneck. If one or more of a device's monitors are above the warning threshold while another device is constrained, this is considered a latent bottleneck. Forecasted bottlenecks The performance analysis algorithm scans for bottlenecks on each system. If no bottlenecks are found for a given system, then performance analysis scans forward, using the forecasted graph. The forecast is of the same length as the report period. For example, a report period of one month can have a forecast of one month into the future. The forecast is used only if no bottlenecks are found in the real data. Only the first bottleneck that is found in the forecast is reported.<br />
<br />
17.7.3 Setting critical and warning threshold values The Report Viewer provides two thresholds, warning (yellow) and critical (red), used to determine quickly which systems exceed preferred levels. These threshold values appear in three places: as red and yellow cells in the Table view, as red and yellow lines on the Hypergraph and on the graphs in the Graph pane, and in the function of performance analysis. Highlighted monitors: As shown in Figure 17-22, several of the monitors listed in the monitor window are highlighted. The threshold settings for these monitors are critical to the optimum function of the performance analysis. If you change the threshold settings for these monitors, the effect on performance analysis will be unpredictable.<br />
<br />
620<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To set the Warning and Critical thresholds, click Monitors tab (Figure 17-22).<br />
<br />
→ Settings, then select the<br />
<br />
Figure 17-22 Threshold settings<br />
<br />
When you are in the Monitors window, you see the monitors listed in the box to the left and the input boxes for the threshold settings to the right. Help for a setting is displayed in the area at the bottom. Click a monitor in the box, then enter a value in the Critical threshold or Warning threshold field. Note: When setting Critical and Warning thresholds for the monitors, some monitor thresholds are expressed as a percentage, and some have an alternative setting, such as Megabytes free or packets/sec. When an alternative setting is available, the box labeled “Show thresholds as percent of maximum value” will be available. Decide which units are most appropriate for your threshold settings, and select or clear the box as appropriate. To return other monitors to their default settings, click the Return to defaults button. Only your currently selected monitor will be reset to its default threshold settings; the other monitors will be unaffected. Repeat for each monitor that you want to return to its default settings.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
621<br />
<br />
17.7.4 Forecast The forecast function is available by clicking the button while viewing the Capacity Manager report. The function allows you to see Capacity Manager's prediction of the future performance of your selected systems. To create its forecast, Capacity Manager uses a linear regression based on a least squares fit with a confidence interval of 95%. For the forecast to be valid, Capacity Manager needs a minimum of 21 days of previously collected data where the System Monitors have been running at least 50% of the time.<br />
<br />
Figure 17-23 Forecast graph<br />
<br />
To see the forecast for your selected systems, click the Forecast icon in the lower-right corner of the window. A graph similar to the one shown in Figure 17-23 displays. The forecast is for the monitor that you currently have selected. To see a forecast for another monitor, click its name in the monitor box. Note: You cannot use both zoom and forecast at the same time; they are mutually exclusive such that one is turned off when the other is turned on. The forecast line is a dashed line with an arrow at the end. The forecast interval is a multiple of your data collection period. The default prediction period is set to the same length as the data collection period. For example, if you have a month of collected data, the forecast will be for a month into the future. The confidence interval is represented by the dotted lines above and below the forecast line. The vertical bar at the beginning of the forecast data depicts the range. The gap between the actual collected data and the beginning of the predicted data serves as a separator between these two data sets.<br />
<br />
622<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Capacity Manager displays one of two warnings if your forecast is not valid. Do not use invalid forecasts to make decisions about your systems. The warning ares: Data collection period too short for a valid forecast. To generate a valid forecast, you need at least 21 days of data. System 'X' does not have enough data for forecasting. or Multiple systems do not have enough data for forecasting. One of these two messages will appear when you have a sufficiently long period for data collection, but one or more monitors were not on for at least 50% of the time during the data collection period. Note: The forecast is more meaningful for individually graphed systems than for those shown in a trend graph. To change your graph from a trend graph to a graph of individual systems, either set your trend graph threshold to a higher number or select fewer systems to graph at one time.<br />
<br />
Chapter 17. Capacity Manager<br />
<br />
623<br />
<br />
624<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Part 5<br />
<br />
Part<br />
<br />
5<br />
<br />
Working with bottlenecks In this part, we show you how to analyze your system to find performance bottlenecks and what to do to eliminate them. We describe an approach you can take to solve a performance bottleneck. We also provide details on what to look for and how to resolve problems. Included is a sample analysis of real-life servers, showing how tools can be used to detect bottlenecks and what the recommendations are for particular systems. This part includes the following chapters: Chapter 18, “Spotting a bottleneck” on page 627 Chapter 19, “Analyzing bottlenecks for servers running Windows” on page 655 Chapter 20, “Analyzing bottlenecks for servers that are running Linux” on page 687 Chapter 21, “Case studies” on page 707<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
625<br />
<br />
626<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
18<br />
<br />
Chapter 18.<br />
<br />
Spotting a bottleneck A bottleneck occurs when any server subsystem prevents the other subsystems from running at peak capacity. This chapter can help you detect a bottleneck problem and show you what to look for so that you will have all the data that you need to identify possible solutions. Having the information that you need can be useful if you are facing a situation where a performance problem is already affecting a server. This type of situation is a reactive situation where you need to follow a series of steps that lead to a concrete solution to restore the server to an acceptable performance level. In addition, over time, experienced that you gain from solving server bottlenecks is very useful when performing new server configuration or server consolidation exercises. There are a number of reasons why you need to fix performance problems, and there is usually a cost that is associated which each of them. To resolve a performance bottleneck problem, you need to be able to answer the following questions: Where is the bottleneck? How can it be fixed? How much will it cost?<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
627<br />
<br />
You can use this chapter as a methodology to spot server performance bottlenecks. In this chapter, we provide a number of worksheets that you can complete based on the performance measurements that you gather from the server. This chapter describes the following steps that we recommend as a bottleneck detection strategy: 1. Know your system. 2. Determine if the bottleneck is real or simply a misunderstood expectation. 3. Back up the system. 4. Monitor and analyze each subsystem during the time the bottleneck is expected to occur. 5. Identify the primary bottleneck and any latent bottlenecks. 6. Fix the cause of the bottleneck by making only one change at a time. 7. Go back to step 4 until you are satisfied with the performance of the system. Tip: You should document each step, the changes that you make, and their affect on performance. The topics that we discuss in this chapter are: <br />
<br />
628<br />
<br />
18.1, “Achieving successful performance tuning” on page 629 18.2, “Step 1: Gathering information” on page 631 18.3, “Step 2: Monitoring the server’s performance” on page 633 18.4, “Step 3: Fixing the bottleneck” on page 651 18.5, “Conclusion” on page 654<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
18.1 Achieving successful performance tuning If you want to increase your success rate and reduce the time that you spend on each case, you must use a repeatable methodology. Here, we list how to achieve successful performance tuning: Perform general maintenance of the server The first job is to ensure that the server is up to date on service packs, patches, and drivers. A good procedure is to perform general maintenance on the server, which includes rebooting the server, defragmenting the drives, and getting the machine up to date on drivers and patches. In some cases, performance bottlenecks are caused by improper service pack, BIOS, or driver configurations or incompatibilities. Ensure that the server is up to date with the latest system software before you waste time chasing bottlenecks. Develop accurate and realistic goals Any performance improvement quest needs to have realistic boundaries. Having unrealistic expectations might lead you to spend unreasonable time and money. Also, do not be afraid to re-evaluate your goals during the process. Research what the expectations are for this server. Does the customer expect the new 8-way server to be twice as fast as the older 4-way? This expectation might or might not be a valid expectation. When the expectation involves comparing one server to another, check published industry-standard benchmarks that are relevant to the production application environment to get a rough idea of how the two servers should compare. Consider that benchmarks are performed by very skilled engineers that know how to extract the most performance from each system. It might not be possible to obtain a similar result in production environments, so err on the conservative side. Also remember that response times can vary widely, depending upon configuration. Throughput is the only accurate way to compare multi-user capacity of two different servers. When a customer says, “Our server is slow,” it can mean many things. Will an improvement of 20% be sufficient? This time of investigation is the perfect time to identify the expectation. If the customer says the server is slow, ask how much faster they need it to be. Document the answer and try to obtain a reasonable expectation before you launch into an extensive bottleneck detection effort. If the customer wants the system to be five times faster, this might not be practical without a server replacement, application modification, or both.<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
629<br />
<br />
Gather relevant background information This information is key to successful bottleneck removal. To gather relevant background information, you should use an iterative process for questioning. A good starting point is to use the questions that we list in 18.2, “Step 1: Gathering information” on page 631. Have a good understanding of the system This step is half of the solution. Do not try to diagnose a complex multi-server bottleneck if you are just learning the basics. Do your homework and ask for expert assistance if needed. Use a methodological approach Time and stress are two frequent parameters during a troubleshooting situation. Just trying this or that will not get you anywhere. Prepare your battle plan before getting to the site, keep it simple, methodical, and consistent. List all necessary performance metrics and counters Having a good understanding of the system and the performance problems should lead you to a set of parameters that can help you resolve the situation. As you understand the problem in more detail, use more specific counters to focus on the problem. Start with the simple counters first, then dig deeper into the bottlenecked component. Gather a baseline of the system’s current performance If you have stated your goals, you need to be able to measure the results of your actions. Without a baseline, you cannot tell if you have met your goals. Validate your interpretation of counter values Try, if possible, to record all counters for the recommended objects that we list in 18.3, “Step 2: Monitoring the server’s performance” on page 633. These counters give you all the data that you need to drill deeper into problems as you learn more about the bottleneck, without having to take multiple traces. Make a permanent record of your progress Document your steps, changes, and results in a dedicated performance notebook. Loose paper tends to disappear into the recycling bin. You will thank yourself in six months when similar problems happen. Over time, this notebook can help you build your bottleneck detection expertise. If in doubt, contact an expert Replacing hardware is costly, not only in terms of parts but also of machine downtime. If you do not feel comfortable about a solution, consult experts. This consultation is where your good understanding of the system and your notebook can prove handy.<br />
<br />
630<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
18.2 Step 1: Gathering information Most likely, the only first-hand information you have are statements such as: “There is a problem with the server.” It is crucial to probe questions to clarify and to document what the problem is. Here is a list of questions that you need to ask to help clarify the problem: Could you give me a complete description of the server in question? – – – – –<br />
<br />
Model Age Configuration Peripheral equipment Operating system<br />
<br />
Can you tell me what the problem is exactly? – What are the symptoms? – Is the number of users on the server now the same as when the server was installed? – Description of any error messages. Some people might have problems answering these questions. Any extra information that you find might give you hints about how to solve the problem. For example, someone might say: “It is really slow when I copy large files to the server.” This information might indicate a network problem or a disk subsystem problem. Keep in mind that people often describe problems by discussing poor response time. Knowing server response time is only a part of the picture. The most important metric for predicting server performance is throughput. For example, how many transactions per second or bytes per second is the server sustaining and how much is needed? Knowing the answers to these questions can help you determine if the server will ever be able to support the required load. No amount of server optimization is going to improve performance to the desired level if the customer needs higher bandwidth than the network can possibly sustain. Who is experiencing the problem? Is one person, one particular group of people, or the entire organization experiencing the problem? This information helps you determine if the problem exists in one particular part of the network, if it is application dependent, and so forth. If only one user is experiencing the problem, then the problem might be the user’s PC. The perception clients have of the server is usually a key factor. From this point of view, performance problem might not be directly related to the server. The network path between the server and the clients could easily be the<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
631<br />
<br />
cause of the problem. This path includes network devices as well as services provided by other servers such as domain controllers. Can the problem be reproduced? All reproducible problems can be solved. If you have sufficient knowledge of the system, you should be able to narrow the problem to its root and decide which actions should be taken. If the problem can be reproduced, you can see the problem and understand it better. Document the sequence of actions necessary to reproduce the problem at any time: – What are the steps necessary to reproduce it? Knowing the steps can let you reproduce the same problem on a different machine under the same conditions. If this works, you have the opportunity to use a machine in a test environment and eliminate the risk of crashing the production server. – Is it an intermittent problem? If the problem is intermittent, the first thing to do is to gather information and find a path to move the problem to the reproducible category. The goal here is to have a scenario to make the problem happen on command. Important: This step is critical. If you cannot reproduce the problem, there is little chance of finding the bottleneck by taking a trace, unless you are extremely lucky. – Does it occur at certain times of the day or certain days of the week? This information might help you determine what is causing the problem. It might occur when everyone arrives at work or returns from lunch. Look for ways to change the timing (that is, make it happen less or more often). If there are ways to do so, the problem becomes a reproducible one. – Is it unusual? If the problem falls into the non-reproducible category, you might conclude that it is the result of extraordinary conditions and classify it as fixed. In real life, there is a high percentage that it will happen again. When did the problem start? Was it gradual or did it occur very quickly? If the performance issue occurred gradually, then it is likely to be a sizing issue, or if it appeared overnight, then the problem could be caused by a change made to the server or peripherals.<br />
<br />
632<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Have any changes been made to the server (minor or major) or are there any changes in the way clients are using the server? Did the customer alter something on the server or peripherals recently that might have caused the problem? Is there a log of all network changes available? Demands could change based on business changes, which could affect demands on servers and network systems. Are there any other servers or hardware components involved? Are there any logs available? What is the priority of the problem? When does it need to be fixed? – Does it need to be fixed in the next few minutes or days? – How massive is the problem? – What is the related cost of the problem?<br />
<br />
18.3 Step 2: Monitoring the server’s performance Important: Before taking any troubleshooting actions, back up all data and the configuration information to prevent a partial or complete loss. At this point, you should begin monitoring the server. The simplest way to monitor the server is to run monitoring tools from the server that is being analyzed. See the appropriate chapter for your operating system: Chapter 14, “Windows tools” on page 471 Chapter 15, “Linux tools” on page 537 Chapter 16, “ESX Server tools” on page 579 Note: The remainder of this chapter applies to each of these operating systems. However, the specific counters are from Performance Monitor in Windows. You need to create a performance log of the server during its peak time of operation (for example 9:00 a.m. to 5:00 p.m.). When creating the log, if available, include a minimum of the following objects: <br />
<br />
Processor System Memory Physical disk Network interface<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
633<br />
<br />
Based on server type, analyze the important subsystems that are likely to be the source of the performance bottleneck. See Chapter 2, “Understanding server types” on page 11 for details about which subsystems are important. Then, complete the information in Table 18-1. Table 18-1 Server type and key subsystems<br />
<br />
Information<br />
<br />
Your details<br />
<br />
Server type Important subsystems (from Chapter 2, “Understanding server types” on page 11)<br />
<br />
1. 2. 3. 4.<br />
<br />
Before you begin, remember that a methodical approach to performance tuning is important. Our recommended process that you can use for your System x server performance tuning method is as follows. 1. Understand the factors affecting server performance. This book helps you to do so. 2. Measure the current performance to create a performance baseline to compare with your future measurements and to identify system bottlenecks. 3. Use the monitoring tools to identify a performance bottleneck. By following the instructions in the next sections, you should be able to narrow the bottleneck to the subsystem level. 4. Improve the component that is causing the bottleneck by performing some actions to improve server performance in response to demands. Note: It is important to understand that you can obtain the greatest gains from upgrading a bottlenecked component when the other components in the server have ample power left to sustain an elevated level of performance.<br />
<br />
634<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 18-1 shows the position of five subsystems (CPU, memory, network, disk, and operating system) represented by a letter from a to e on a performance scale. A well-optimized system has all of its subsystems grouped together. This figure has one component (a) which represents the system bottleneck. Component a needs to be moved closer to the other subsystems. Lower Performance<br />
<br />
Higher Performance<br />
<br />
a<br />
<br />
b cde<br />
<br />
a, b, c, d, e, represent the performance capabilities of each subsystem component:CPU, Memory, Network, Disk and OS<br />
<br />
Figure 18-1 System with one primary bottleneck (a) but well-balanced other subsystems<br />
<br />
In addition, ensure that other components in the server are not latent bottlenecks working just below the utilization of the bottlenecked component. Latent bottlenecks limit improvements that are realized by any upgrade. In general, components that have average utilization between 60% to 70% are likely to be latent bottlenecks. (We provide definitions of the different types of bottlenecks in 17.7.2, “Types of bottlenecks” on page 620.) If there are latent bottlenecks in a system, then you must reconfigure or upgrade both the primary component that is causing the bottleneck and the component that has a latent bottleneck to obtain optimal performance. Figure 18-2 shows one primary (component a) and one latent bottleneck (component b). Lower Performance<br />
<br />
a<br />
<br />
Higher Performance<br />
<br />
b<br />
<br />
cde<br />
<br />
a, b, c, d, e, represent the performance capabilities of each subsystem component:CPU, Memory, Network, Disk and OS<br />
<br />
Figure 18-2 System with a primary (a) and a latent (b) bottlenecks<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
635<br />
<br />
In this case, upgrading component a moves the primary bottleneck to component b, as shown in Figure 18-3. If several components are causing latent bottlenecks, perhaps the most cost-effective solution is to replace the entire server. Lower Performance<br />
<br />
a<br />
<br />
Higher Performance<br />
<br />
b<br />
<br />
cde<br />
<br />
a, b, c, d, e, represent the performance capabilities of each subsystem component:CPU, Memory, Network, Disk and OS<br />
<br />
Figure 18-3 Moving the bottleneck from (a) to the latent bottleneck (b)<br />
<br />
5. Measure the new performance so that you can compare the performance before and after the tuning steps. Performance tuning is a continuous process, so you must maintain ample record-keeping to help analyze future demands. In this way, you can predict and eliminate problems before they even occur. Analysis tools, such as System Monitor in Windows, give access to all the listed objects and counters in the remaining tables in this chapter. The number of objects and counters that you measure and how often you measure them depends on two questions: What are you trying to detect? What kind of disk resources do you have available? The sampling period, or the amount of time between data collection points, always comes with a trade-off. The smaller the sampling period, the bigger the file will be. On the other hand, an extremely long sampling period will not be suited to detecting temporary peaks in the system utilization. For example, if you record all the counters listed in this section, you could expect a log file size for a one-second sampling rate to be close to 30 MB per hour of recording. Tip: For an 8-hour trace, a sample time of 60 seconds or longer is sufficient to diagnose consistently slow server performance.<br />
<br />
636<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Keep in mind that the techniques recommended in this section do not require high-resolution sampling unless the problem you are diagnosing only occurs for a brief period of time. If the problem is a consistently slow server, then use sample times that result in manageable sized trace files. If space is a problem and you want to record for a long period of time, you should use the circular logging option in the Performance console. Using the alerts that we explain in 14.1.4, “Using performance logs and alerts” on page 486 can also help you. Sampling can also be a two-step action: first, record using only core counters from the different subsystems until you can narrow down the search to one or two subsystems. After that, you can increase the number of counters in one subsystem and decrease the sampling rate. The obvious downside to this method is the extra work and inconvenience of taking multiple traces.<br />
<br />
18.3.1 Where to start Now we have a good idea about what we want to accomplish, but how do we proceed? Which counters should we use and how do we know when a bottleneck occurs? These are the crucial steps which cause many people to stumble, because there are an enormous number of complex performance objects and associated counters. However, detecting server subsystem bottlenecks is quite easy after you understand the primary subsystems and the primary subsystem counters that you can use to diagnose the health of each subsystem. Our strategy for bottleneck detection uses a top-down approach. In a top-down approach, we take a high-level look at each of the server's primary subsystems by examining each of the primary counters that indicate bottlenecks are present. We want to first validate the health of each primary server subsystem. The best way to determine if a subsystem is performing poorly is to identify the primary counters and the corresponding thresholds that can be used to identify performance bottlenecks for that subsystem. Then we examine each of the primary counters and compare them to the thresholds to know if the subsystem is healthy or unhealthy. These primary performance counters are critical, because you can use them as a pass/fail test to determine the health of a subsystem. Only after a subsystem has failed the primary counter test do you need to perform more extensive analysis. This pass/fail testing makes bottleneck detection much easier, because you can avoid all the complex counters for any subsystem that passes the primary counter test because that system is healthy.<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
637<br />
<br />
Table 18-2 lists the primary performance objects and associated counters along with the corresponding threshold for each of the server subsystems that we used for our pass/fail test (in order of most likely to cause a bottleneck). Table 18-2 Primary performance objects<br />
<br />
Subsystem<br />
<br />
Counter<br />
<br />
Guidance<br />
<br />
Disk<br />
<br />
Physical Disk: Avg. Disk sec/Transfer<br />
<br />
Must be lower than about 25 ms.<br />
<br />
Memory<br />
<br />
Memory: Available Bytes<br />
<br />
Should be no lower than 20% to 25% of installed memory.<br />
<br />
Memory<br />
<br />
Memory: Page Reads/sec<br />
<br />
Ideally should be zero, but sometimes this is not possible because some applications, such as Lotus Domino and SAP, use the page file for memory mapped file communication. In any event, the combined value of this counter and Page/Writes/sec should not be higher than about 150 I/Os per second per disk used in the page file device.<br />
<br />
Memory<br />
<br />
Memory: Page Writes/sec<br />
<br />
Ideally should be zero, but sometimes this is not possible because some applications such as Lotus Notes and SAP, use the page file for memory mapped file communication. In any event, the combined value of this counter and Page/Reads/sec should not be higher than about 150 I/Os per second per disk used in the page file device.<br />
<br />
Processor<br />
<br />
Processor: % Processor Time_Total<br />
<br />
Should be lower than about 70% to 80%.<br />
<br />
Processor<br />
<br />
Processor: % Processor Time_(N)<br />
<br />
Each processor should be lower than about 70% to 80%.<br />
<br />
Network<br />
<br />
Network Interface: Bytes Total/sec<br />
<br />
Should be lower than about 50% to 60% of maximum sustainable bandwidth. For Gigabit Ethernet in a Xeon MP or Xeon DP-based system, this is about 70-80 MBps. For Gigabit in a Pentium III-based system, this is about 30-40 MBps. Use 1/10 of these values for 100 Mbps Ethernet.<br />
<br />
638<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Your value<br />
<br />
Subsystem<br />
<br />
Counter<br />
<br />
Guidance<br />
<br />
Your value<br />
<br />
Network<br />
<br />
Network Interface: Packets/sec<br />
<br />
Should be no higher than about 70 000-80 000 packets per second for Gigabit Ethernet in Xeon MP or Xeon DP-based servers. Should be no higher than about 30 000-40 000 packets/sec for Gigabit adapters in Pentium III-based servers. For 100 Mbps Ethernet, use 1/10 of these values.<br />
<br />
Just to reiterate, our top-down approach is to perform a pass/fail test for each of the primary counters listed in Table 18-2. This pass/fail testing provides a simple, objective way to determine systemically if each subsystem is healthy. Only after you find one or more unhealthy subsystems do you then drill down deeper to learn more about the bottleneck. If you find bottlenecks, you can go to Chapter 19, “Analyzing bottlenecks for servers running Windows” on page 655 to determine if you relieve the bottleneck by upgrading or tuning the specific subsystem. After tracing hundreds of server configurations, we have learned that the most likely server hardware components to cause a bottleneck are, in order: Disk subsystem Memory subsystem Disk and memory technologies have lagged significantly behind the performance curve of processors and network technology and this is one reason why disk and memory are the two most frequently found server bottlenecks. However, equally important is that many administrators configure server disk subsystems based solely on capacity requirements. Often this limitation means purchasing fewer, higher-capacity disks. Thus, here are fewer disk heads to service the required I/O data rates. Because the disk subsystem is so often the bottleneck, let us start there.<br />
<br />
18.3.2 Disk subsystem The disk subsystem is comprised of the disk controller, its device driver, the SCSI bus or Fibre Channel bus that connects the system to the disks, and finally the individual disk drives. One key point to understand about disk subsystems is that for most commercial server workloads, physical disk I/O is almost always random. Servers provide data storage for the entire population of network attached users connected to that server. Each user is requesting different data from different locations on the disks of the server. The server is trying to cache data in buffers in memory, and the disk controller is trying to cache data in<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
639<br />
<br />
controller cache. The OS is also using a file system to store data that becomes fragmented over time. So by the time a disk I/O actually reaches the disk drive, it is almost always to a very different address than the previous disk I/O request. This means the disk controller has to process the I/O command, send the command to the disk, move the head to the new data track (seek), wait for data to rotate underneath the head (rotational latency), read or write the data, and send a completion status back to the controller to notify the OS that the I/O is complete. Even if server users are executing applications that are performing sequential I/O, because of all the caching and disk fragmentation, much of the physical I/O at the disk drive can still be random. Most commercial applications do not perform sequential I/O. Databases, e-mail, file serving, and most multi-user commercial applications generate random disk I/O which introduce seek and rotational latency delays for nearly every disk access, greatly reducing sustained throughput. Thus, we should not expect disks to generate very high data rates for most commercial servers. Of course, there are always exceptions. For example, High Performance Computing (HPC) servers might run a single process that reads a large array of data from disk, and then writes a large solution set of data back to disk. In this case, a single process will be accessing the disk, not a large number of concurrent users generating multiple unique disk I/O requests. Furthermore, HPC workloads tend to read disk data using very large disk I/O sizes, thereby increasing the sustained disk I/O bandwidth. For HPC workloads, it is possible to saturate PCI, SCSI, or Fibre Channel bandwidth limitations, but in most commercial workloads, seek and latency operations dominate the I/O time and significantly lower sustained I/O rates. A closer look at the time to perform disk I/O operations provides us with a critical understanding of how to avoid and diagnose disk subsystem bottlenecks for commercial workloads (random I/O). Let us take a look at a typical high-speed 15 000 RPM disk drive. For this disk, the total access time can be calculated as:<br />
<br />
640<br />
<br />
Average seek<br />
<br />
3.8 ms<br />
<br />
Average latency<br />
<br />
2.0 ms<br />
<br />
Command and data transfer<br />
<br />
< 1 ms<br />
<br />
Average random access time:<br />
<br />
6.8 ms per operation (147 operations/sec)<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
It takes about 0.0068 seconds for a 15 000 RPM disk to perform an average disk operation. Therefore, in one second the disk can only do about 147 I/Os per second. This is calculated as 1 / 0.0068sec = 147 I/Os per second per disk. Because most commercial applications are accessing data on disk in 4, 8, or 16 KB sizes, the average bandwidth sustained by a disk drive can easily be calculated. For example, at 8 KB I/O size: 8 KB per I/O * 147 I/Os per second = 1.15 MBps per disk At about 1.15 MBps per disk, it takes a lot of disks to start to stress the PCI bus (100 MHz PCI-X can handle 800 MBps), the SCSI bus (Ultra 320 is 320 MBps), or Fibre Channel (2 Gbit = 200 MBps full duplex). Far too many people are concerned about PCI bus and SCSI bus configuration, when in fact they are usually no cause for worry. In general, the number of disk drives, disk fragmentation, RAID strategy, and the ability of the application to queue a large number of disk I/O commands to the physical array are the leading causes of commercial server disk subsystem bottlenecks. For 10 000 RPM disks, the same calculations can be: Average seek<br />
<br />
4.9 ms<br />
<br />
Average latency<br />
<br />
3.0 ms<br />
<br />
Command and data transfer<br />
<br />
< 1 ms<br />
<br />
Average random access time:<br />
<br />
8.9 ms per operation (112 operations/sec)<br />
<br />
8.9 ms corresponds to 1/ 0.0089 = 112 I/O operations per second, and at 8 KB per I/O, a 10 000 RPM disk can sustain only about: 8 KB per I/O * 112 I/Os per second = 896 KBps per disk (» 900 KBps). Now, it might be tempting to use the I/O rates that we just calculated as indicators for disk performance bottlenecks, but this would be a grave mistake because I/O rates can vary wildly. For example: In some special cases, disks perform sequential I/O. When this occurs, seek and rotational latency will be zero or near-zero, and disk I/O rates will increase dramatically. Even though this is rare, we do not want to use a performance indicator that works only some of the time. Our calculations assume average seek and latency times. Drive vendors produce average seek times from 1/3 track-seek range measurements. Full track seek times are much longer, and disks that are accessing more than 1/3 of capacity will have longer seek times and significantly lower sustained I/O rates.<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
641<br />
<br />
RAID strategy affects the number of physical I/Os a disk will actually perform. A random write to a RAID-5 disk array will generally produce two read and two write disk operations to the RAID-5 array, but operating system disk counters count this as one disk I/O. Stripe size and I/O request size will affect the number of physical disk operations performed. For example, a very large disk read or write operation for 64 KB in size sent to an array that is using 4 KB stripe size will generate 16 physical disk I/O operations. But the OS disk counters will count this as one disk I/O because the OS does not know anything about the stripe size the disk array controller is using. Clearly sustained disk I/O rates can vary greatly. So we do not recommend using average disk I/O rates as an indicator of a disk bottleneck unless you thoroughly understand the disk workload and storage configuration. A better way to identify disk bottlenecks is to use our understanding of disk operation combined with average response time. We know a 15 000 RPM disk requires about 7 ms of average disk access time and a 10 000 RPM disk has about 9 ms of disk access time. We can use this information to greatly simplify disk bottleneck detection. However, before we launch into that discussion, we need to discuss one more characteristic of disk drive operation: optimization. Modern disk drives can actually increase the sustained throughput when given more work to do. If multiple read and write operations are sent to a disk drive, it can use elevator seek optimization and rotational positioning optimization to reorder the physical I/Os to increase the sustained I/O rate as compared to when processing a single disk read or write operation at a time. By sending two or more disk commands to the disk, it can reorder the operations to reduce the amount of seek time and even rotational latency. However, even more significantly, when a seek is occurring for an existing I/O request, and another disk I/O command arrives, the disk can determine if it can access that data while performing the current seek command. That is, while a long seek operation is occurring, the processor on the disk determines if the head will pass over any of the data addresses for read or write commands that just arrived in its queue after the long seek was started. If so, the read or write command in the queue will be accomplished while the head is moving out to the track for the original I/O. The key message from the discussion of disk I/O operation is this: disk drives perform best and have optimal throughput when given two to three (no more than three) disk operations at the same time.<br />
<br />
642<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
As a rule of thumb, the optimal response time of a disk drive is about 2.5 times the normal access time. For 15 000 RPM disk drives, this is about 17 ms For 10 000 RPM disk drives, the optimal response time is about 22 ms For bottleneck detection purposes, use a range of values instead of a precise number. A good rule of thumb to use is that a disk subsystem is healthy whenever the disk subsystem is performing read and write operations with less than about 20-25 ms per I/O. When the average disk latency is much greater than 25 ms, then the disk subsystem can be considered unhealthy and is a bottleneck. Rule of thumb: When Avg. Disk Seconds/Transfer (the disk latency counter) is significantly greater than 25 ms, the disk subsystem is unhealthy and is a bottleneck. Remember, this counter does not tell us how to fix the problem, it only indicates there is a problem. You can look at one simple Performance Monitor counter and know if your disk subsystem is healthy or not. Simply look at Avg. Disk Sec/Transfer for each physical disk drive or array in your server. Use the chart mode to identify the peaks, and if this counter spends a significant amount of time over 25 ms during the period where the server is considered to have a bottleneck, you can consider your disk subsystem unhealthy. Do not order a new SAN if your average disk latency is 26 ms. Clearly, there is a range of latencies where performance will be acceptable. Each server administrator must decide when to consider the average latency too great. Some server administrators will use 30, 40 or 50 ms, others will want ultimate performance and take action at 25 ms. However, if the disk subsystem is running at 60 or 80 ms on a regular basis then the disk subsystem is clearly slowing down server performance. On many occasions, we have seen overloaded disk subsystems performing with 1 or 2 seconds of average latency (1000 or 2000 ms). This is a very slow disk subsystem. When performing clustering, the threshold is a little higher. Clustering solutions with SCSI require the disk subsystem to disable write-back mode. When RAID-5 is used in write-through mode, the server must perform two reads and two writes for each and every write command. In this case, writes can take twice as long as RAID-0 or RAID-1. So for clustering solutions, or in any case where the server is performing a large amount of write operations for RAID-5 with write-through disk controller settings, we want to use about 40-50 ms as our threshold for identifying disk performance bottlenecks.<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
643<br />
<br />
After you have identified the disk subsystem as unhealthy, Chapter 19, “Analyzing bottlenecks for servers running Windows” on page 655 can help you understand how to improve the performance of an unhealthy disk subsystem. Complete Table 18-3 with your results. When you are done, examine the primary disk counter in the table to determine if the disk subsystem is a bottleneck. If so, refer to 19.4.1, “Analyzing disk bottlenecks” on page 669 for more details regarding how to analyze and resolve the disk bottleneck. Table 18-3 Performance console counters for detecting disk bottlenecks<br />
<br />
Counter<br />
<br />
Is a bottleneck if...<br />
<br />
Physical Disk: Avg. Disk sec/Transfer<br />
<br />
This is the primary counter for detecting disk bottlenecks. This is the time to complete a disk I/O. For optimal performance, this should be less than 25 ms. Consistently running over .025 s (25 ms) indicates disk congestion. Note: Examine the counters for physical disk not logical disk.<br />
<br />
Physical Disk: Avg. Disk Queue Length<br />
<br />
This counter is useful to know how many disks to use in a RAID Array. In general, optimal performance is obtained when this counter averages 2-3 times the number of disks in each physical array. A high number indicates queuing at the physical volume. This is bad in that it increases response time and degrades performance. A high number is also good because it indicates the application I/O workload will scale simply by adding disks to the array.<br />
<br />
Physical Disk: Avg. Disk Bytes/Transfer<br />
<br />
This is the average number of bytes transferred to or from the disk during write or read operations. You should also compare this value against the stripe size of the RAID array (if you are using hardware-based RAID). We recommend you configure the stripe size to be at least equal to long-term average value of this counter. For example, if the Avg. Disk Bytes/Transfer is 4 KB then use an 8 KB stripe size on the RAID array volume.<br />
<br />
Physical Disk: Disk Bytes/sec<br />
<br />
Sum this counter's value for each disk drive attached to the same SCSI/Fibre Channel controller and compare it to 70% to 80% of the theoretical throughput. If these two numbers are close, the bus is becoming the disk subsystem's bottleneck. Review the disk subsystem data path.<br />
<br />
Physical Disk: Split IO/sec<br />
<br />
A split I/O is a result of two different situations: the data requested is too large to fit in one I/O or the disk is fragmented.<br />
<br />
644<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Your result<br />
<br />
18.3.3 Memory subsystem Most memory counters are related to virtual memory management. Virtual memory counters will not help identify if the server has insufficient physical memory capacity and is running poorly as a result of excessive disk paging. However, there are a few counters that can help us determine if the memory configuration is healthy. Complete Table 18-4 by monitoring the memory counters on your server. Then examine the primary memory counters to determine if the memory capacity is causing a bottleneck. If so, refer to 19.3, “Analyzing memory bottlenecks” on page 661 for more details regarding these counters. Table 18-4 Performance console counters for detecting memory bottlenecks<br />
<br />
Counter<br />
<br />
Is a bottleneck if...<br />
<br />
Your result<br />
<br />
Memory: Page Reads/sec<br />
<br />
This is a primary memory bottleneck counter. Ideally, this value should be close to zero. However, sometimes it is not possible to eliminate paging because some applications (such as Lotus Domino) use the page file for communication between processes. However, if paging is so high as to saturate the page disk device then performance will suffer. If this counter is consistently higher than 150 I/Os per disk per second for the paging device, the server has a memory or paging device bottleneck. Page Reads/sec is the rate at which the disk was read to resolve hard page faults. It shows the number of read operations, without regard to the number of pages retrieved in each operation. Hard page faults occur when a process references a page in virtual memory that is not in a working set or elsewhere in physical memory, and must be retrieved from disk. This counter is a primary indicator of the kinds of faults that cause system-wide delays. It includes read operations to satisfy faults in the file system cache (usually requested by applications) and in non-cached mapped memory files. Compare the value of Memory:Pages Reads/sec to the value of Memory:Pages Input/sec to determine the average number of pages read during each operation.<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
645<br />
<br />
Counter<br />
<br />
Is a bottleneck if...<br />
<br />
Memory: Page Writes/sec<br />
<br />
This is a primary memory bottleneck counter. Ideally, this value should be close to zero, but sometimes it is not possible to eliminate paging because some applications use the page file for communication between processes. However, if paging is so high as to saturate the page disk device then performance will suffer. If this counter is consistently higher than 150 I/Os per disk per second for the paging device, the server has a memory or paging device bottleneck. Page Writes/sec is the rate at which pages are written to disk to free up space in physical memory. Pages are written to disk only if they are changed while in physical memory, so they are likely to hold data, not code. This counter shows write operations, without regard to the number of pages written in each operation. This counter displays the difference between the values observed in the last two samples, divided by the duration of the sample interval.<br />
<br />
Memory: Available MB<br />
<br />
This is a primary memory bottleneck counter.<br />
<br />
Memory: Pool Nonpaged Bytes<br />
<br />
This indicates the amount of RAM in the non-paged pool system memory area where space is acquired by operating system components as they accomplish their tasks. If this value has a steady increase without a corresponding increase in activity on the server, it might indicate that a process that is running has a memory leak, and should be monitored closely.<br />
<br />
Paging File: % Usage Peak<br />
<br />
This is a bottleneck if the value consistently reaches 90%.<br />
<br />
Server: Pool Nonpaged Peek<br />
<br />
This is the maximum number of bytes of nonpaged pool the server has had in use at any one point. It indicates how much physical memory the computer should have. Add 20% to this value to determine the amount of installed memory that the server should require.<br />
<br />
Server: Pool Nonpaged Failures<br />
<br />
This is the number of times allocations from nonpaged pool have failed. It indicates that the computer's physical memory is too small. If this value is not zero on a regular basis, the system needs more memory.<br />
<br />
646<br />
<br />
If this value stays at less than 20% to 25% of installed RAM, consider adding memory to the server.<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Your result<br />
<br />
18.3.4 Processor subsystem Determining processor bottlenecks is easy. If the %Processor Time Total of any single processor is sustaining over 70% to 80% utilization, it should be considered a bottleneck. Complete Table 18-5, then examine the primary processor counters to determine if the CPU is a bottleneck. If so, refer to 19.2.1, “Finding CPU bottlenecks” on page 657 for more details regarding these counters and how to eliminate processor bottlenecks. Table 18-5 Performance console counters for detecting CPU bottlenecks<br />
<br />
Counter<br />
<br />
Is a bottleneck if...<br />
<br />
Your result<br />
<br />
Processor:% Processor Time<br />
<br />
This is the primary counter for detecting processor bottlenecks. The CPU is a bottleneck if the value is consistently running over 70% to 80% (excluding any processes that are running in the background at low priority, absorbing all spare CPU cycles). Note: Examine the counters for each CPU installed as well as the _Total value to ensure there are no problems with “unbalanced” or processor affinitized applications.<br />
<br />
Processor: % Privileged Time<br />
<br />
This counter can be used (excluding any processes that are running in the background at low priority, absorbing all spare CPU cycles) to identify abnormally high kernel time and might indicate I/O driver problems Note: Examine the counters for each CPU installed as well as the _Total value to ensure there are no problems with “unbalanced” applications.<br />
<br />
Processor: % User Time<br />
<br />
%User time represents the time spent by the user application on the server. This is an important counter because it shows a breakdown of how the server application is utilizing all the processors. Note: Examine the counters for each CPU installed as well as the _Total value to ensure there are no problems with “unbalanced” application usage. Often, when applications do not scale, they will not start user threads for all the processors in the server. Processors 0, 1 running high % User Time while processors 2, 3, and so forth, are running much lower %User Time indicates insufficient application threading.<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
647<br />
<br />
Counter<br />
<br />
Is a bottleneck if...<br />
<br />
System: Processor Queue Length<br />
<br />
A sustained queue of over much more than four times the number of processors installed indicates a processor congestion.<br />
<br />
Your result<br />
<br />
The good news is that an application that drives the processor queue to a long length can take advantage of a large number of processors. When the server is running at above 75% CPU utilization, check this counter to see if the average queue length is significantly greater than two times the number of installed processors. If so, that application will scale as additional processors are added, up to the point where the Avg. Queue Length is equal to 2N, where N is the number of processors. You always want to see at least two threads per processor. Ideally, two times the number of processors is the optimal queue length when Hyper-Threading is enabled.) Processor: Interrupts/sec<br />
<br />
Processor Interrupts/sec should no longer be used as a simple indicator of performance. Modern device drivers have dynamic mechanisms and batch interrupts when the system is busy, doing more work per interrupt. This causes the number of interrupts per second to be dynamic. When a system is moderately busy, it might require an interrupt to process each LAN or DISK I/O operation. In this mode, the server will have a fairly high interrupt rate. However, as the server becomes busier, multiple disk and LAN operations will be sent under one interrupt request, lowering the interrupt rate and improving interrupt efficiency. In summary, do not use this counter unless you have detailed information about the specific device drivers used in your server.<br />
<br />
648<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
No need to measure; this is not normally used to analyze bottlenecks.<br />
<br />
18.3.5 Network subsystem The network itself can be difficult to analyze. This is because the performance counters captured on the server represent only the load in the network that is destined to or from the particular server where the counters were captured. The counters do not reflect the entire load in the network, which could be causing a serious bottleneck for users connected to the same network as they try to communicate with the server. Unfortunately, the only way to identify network bottlenecks which are not related to the server where the performance counters were monitored is to use a Network Analyzer. This, however, is beyond the scope of this publication. The network subsystem counters can, however, be used to successfully diagnose network bottlenecks caused by excessive traffic to or from a particular server. This excessive traffic will manifest itself as a choke point at the network adapter. Network adapters can have two types of bottlenecks: When the sustainable throughput of the network adapter is reached When the maximum sustainable packet rate of the network adapter is reached In general, these values vary with network adapter type and system configuration, so take these values with a grain of salt. However, in general, the counter Bytes Total/sec should be lower than about 50% to 60% of maximum sustainable bandwidth. This means the following values: <br />
<br />
For Gigabit Ethernet on a Xeon MP/DP-based server: 70-80 MBps For 100 Mbps Ethernet on a Xeon MP/DP-based server: 7-8 MBps For Gigabit Ethernet in PIII-based systems: 30-40 MBps For 100 Mbps Ethernet in PIII-based systems: 3-4 MBps<br />
<br />
Packets/sec rates should be no higher than: For Gigabit Ethernet on a Xeon MP/DP-based server: 70 000 to 80 000 pkts/sec For 100 Mbps Ethernet on a Xeon MP/DP-based server: 7 000 to 8 000 pkts/secs For Gigabit Ethernet in PIII-based systems: 30 000 to 40 000 pkts/sec For 100 Mbps Ethernet in PIII-based systems: 3 000 to 4 000 pkts/sec Complete all fields in Table 18-6. Compare the primary network counters with the sustained thresholds listed to determine if the network subsystem is the bottleneck. If so, refer to 19.5.1, “Finding network bottlenecks” on page 673 for more details about how to resolve network bottlenecks.<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
649<br />
<br />
Note: With Windows 2000 Server, you need to install various networking functions before you can perform this analysis. You need to load the Network Monitor Driver and SNMP services. To monitor network-specific objects in Windows 2000, you need to install the Network Monitor Driver (this is not necessary for Windows 2003): 1. 2. 3. 4. 5. 6. 7. 8.<br />
<br />
Open Network and Dial-up Connections in the Control Panel. Select any connection. Click File → Properties. In the General tab, click Install. Select Protocol. Click Add. Select Network Monitor Driver. Click OK then Close.<br />
<br />
Note: As per Microsoft Knowledge Base entry Q253790, the Network Segment object has been removed from Windows 2000. Table 18-6 Performance console counters for detecting network bottlenecks<br />
<br />
Counter<br />
<br />
Is a bottleneck if...<br />
<br />
Network Interface: Bytes Total/sec<br />
<br />
This is a network subsystem primary counter.<br />
<br />
Network Interface: Bytes Received/sec<br />
<br />
This is a network subsystem primary counter.<br />
<br />
650<br />
<br />
Sustained values over 50% to 60% of the adapter’s available bandwidth indicate a bottleneck. So for Gigabit Ethernet in a Xeon MP/DP-based system, this is about 70-80 MBps. For Gigabit Ethernet in PIII-based systems, this is about 30-40 MBps. Use 10% of these values for 100 Mbps Ethernet.<br />
<br />
Sustained values over 50% to 60% of maximum sustained throughput in the receive direction should be investigated by a network administrator to determine if the network is a bottleneck. Most Gigabit Ethernet adapters can sustain about 800 Mbps in the receive direction. Maximum sustained throughput for 100 Mbps Ethernet in the receive direction is about 80 Mbps.<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Your result<br />
<br />
Counter<br />
<br />
Is a bottleneck if...<br />
<br />
Your result<br />
<br />
Network Interface: Bytes Sent/sec<br />
<br />
This is a network subsystem primary counter. Sustained values over 50% to 60% of maximum sustained throughput in the send direction should be investigated by a network administrator to determine if the network is a bottleneck. Most Gigabit Ethernet adapters can sustain about 800 Mbps in the send direction. Maximum sustained throughput for 100 Mbps Ethernet in the send direction is about 80 Mbps.<br />
<br />
Network Interface: Packets/sec and<br />
<br />
Network Interface: Packets Sent/sec and<br />
<br />
Network Interface: Packets Received/sec<br />
<br />
These are network subsystem primary counters. Packets/sec rates should be no higher than about 50% to 60% of maximum packet/sec rates listed in Table 19-5 on page 676 and Table 19-6 on page 677. Use the Bytes/Total/sec counter value divided by Total Packet/sec counter value to calculate average Bytes per Packet or average packet size for your server workload. Then, using the calculated average packet size (bytes/packet), use Table 19-5 on page 676 and Table 19-6 on page 677 to determine the maximum packet/sec rate sustainable for the adapter speed being used. We recommend detailed network analysis If the sustained values are 50% or greater than the values listed in the table.<br />
<br />
18.4 Step 3: Fixing the bottleneck After you determine which subsystem is the bottleneck, you should examine the options for solving the problem. We discuss these in the next three chapters. Depending on your specific situation, these options could include: CPU bottleneck: – Add more processors – Switch to processors with larger L2 cache – Replace existing processors with faster ones Memory bottleneck: – Add memory<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
651<br />
<br />
Disk bottleneck: – – – – – – – –<br />
<br />
Spread the I/O activity across drives or RAID arrays (logs, page file, etc.) Add disks to the RAID array Use RAID-1 instead of RAID-5 or instead of single disks Correct the stripe size used to match the I/O transfer size Use faster disks Add another RAID controller/channel or Fibre Channel host adapter For Fibre Channel, add a second RAID controller module If running in Write Back (WB) mode, then select Write Through (WT) as a temporary fix if additional drives are not available. Selecting WT whenever Avg. Disk sec/Transfer exceeds 15-18 ms can yield a 20% to 30% increase in throughput for heavily loaded disk configurations compared to WB mode.<br />
<br />
Network bottleneck: – Ensure network card configuration matches router and switch configurations (for example, frame size) – Modify how your subnets are organized – Use faster network cards – Add network cards When attempting to fix a performance problem, remember the following: Take measurements before you upgrade or modify anything so that you can tell if the change had any effect (that is, take baseline measurements). Examine the options that involve reconfiguring existing hardware, not just those that involve adding new hardware. After you upgrade a specific subsystem, other latent bottlenecks might appear in other subsystems. Follow the steps in the flowchart shown in Figure 18-4 as a first step to resolving performance problems.<br />
<br />
652<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Start OS<br />
<br />
CPU<br />
<br />
Tune OS and Application<br />
<br />
Consistently Exceeds 80% Utilization?<br />
<br />
Yes<br />
<br />
Continue until there are no bottlenecks found.<br />
<br />
Upgrade/Add CPU(s) Add Server<br />
<br />
No<br />
<br />
Memory<br />
<br />
Sustained Paging? Mem Pages/sec > 200?<br />
<br />
Yes<br />
<br />
Increase Memory<br />
<br />
No<br />
<br />
Disk<br />
<br />
Average Queue Length Greater Than 2?<br />
<br />
Upgrade Disks Increase Memory Adjust Stripe<br />
<br />
Yes<br />
<br />
No<br />
<br />
Network<br />
<br />
Bytes/Sec Greater than 60% of Network Capacity Sustained?<br />
<br />
Yes<br />
<br />
Upgrade Network Load Balance/ Subnet<br />
<br />
No<br />
<br />
End<br />
<br />
Figure 18-4 Bottleneck flowchart<br />
<br />
Chapter 18. Spotting a bottleneck<br />
<br />
653<br />
<br />
18.5 Conclusion This chapter provides a general approach that you should take when determining server bottlenecks. When trying to find bottlenecks, you should also take into consideration the type of server you are monitoring and what subsystems are potential bottlenecks. Review the chapters in Part 6, “Applications” on page 743 of this book for performance tips on key server applications. Before making any recommendation for a server: Make sure you understand what is causing the bottleneck. Research your recommendations, and be sure what you are proposing will improve server performance. Know how much the upgrade/reconfiguration will cost. It is also good practice to install IBM Director with Capacity Manager for ongoing and proactive analysis of the server. See Chapter 17, “Capacity Manager” on page 591 for information about this tool.<br />
<br />
654<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
19<br />
<br />
Chapter 19.<br />
<br />
Analyzing bottlenecks for servers running Windows This chapter discusses how to use the System Monitor1 console, the Windows Server 2003 built-in tool for monitoring server performance and using its output to analyze server subsystem bottlenecks. How you resolve server bottlenecks depends primarily on your bottleneck detection analysis and findings. Topics that we discuss here are: <br />
<br />
1<br />
<br />
19.1, “Introduction” on page 656 19.2, “CPU bottlenecks” on page 656 19.3, “Analyzing memory bottlenecks” on page 661 19.4, “Disk bottlenecks” on page 667 19.5, “Network bottlenecks” on page 672<br />
<br />
Product screen captures and content are reprinted with permission from Microsoft Corporation.<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
655<br />
<br />
19.1 Introduction Consider the following statements when analyzing Windows 2003 performance: In general, the most frequently found hardware bottlenecks in Windows Server 2003 servers are caused by the disk subsystem and available memory capacity. Often disk bottlenecks can result from too little available memory. Also, when you find disk bottlenecks, remember that it is far easier to add memory than to reconfigure the disk subsystem. So, you need to explore adding memory when you discover disk bottlenecks. When you rule out disk subsystem and memory shortages, the processor and network subsystems are likely the next sources for contention. In general, you can achieve the greatest gains with Windows 2003 system performance by tuning and sizing the memory subsystem, disk subsystem, processor configuration, and network subsystem properly, in that order.<br />
<br />
19.2 CPU bottlenecks The CPU is one of the first components of which most people think when they have a performance problem. All server operations, including all requests from network attached clients, are processed by the server's CPU. For servers whose primary role is that of an application or database server, the CPU is clearly important. However, there is a common misconception that the CPU is the most important part of the server and can be the single measure when comparing system performance. Unfortunately, in practice, this fact is often not the case. These days, CPUs are so fast that their power far exceeds the other server components, especially when the server is configured improperly. Servers are often over-configured with CPU and under-configured with disks, memory, and network components. Remember, CPU performance has increased significantly in recent years, and the CPU is usually only a bottleneck when memory, network, and disk subsystems are performing without any bottlenecks. Bottlenecks in any of these other subsystems means the CPUs must wait, which results in lower CPU utilization. Therefore, it usually pays to check the performance of the other subsystems before upgrading or adding additional CPUs to a poor-performing server. Also consider that the latest processors have evolved to support multiple concurrent threads. Hyper-Threading is one technique to better use the fast Xeon processor core by multiplexing two software threads to run on the same processor core. This Hyper-Threading ensures that when one thread is waiting<br />
<br />
656<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
on slower resources such as memory or I/O, the other thread can execute and keep the processor core busier. However, this dual-thread execution methodology shifts the burden onto the software to create twice as many threads as it did before Hyper-Threading was introduced. In many cases, applications have not evolved to take advantage of this greater parallelism and the additional threads are not always generated. In this case, Hyper-Threading might not yield a significant improvement in performance. Also consider this same threading issue when migrating older applications to a server with a greater number of processors, especially when doing so to gain increased performance. When adding more processors to the system, an application must detect the additional processors and must introduce more threads to take advantage of the increased parallel execution capabilities. This process does not happen automatically, and older applications do not always detect addition processors. The result is often that the existing applications run no faster on the new server with its greater number of processors than they did on the older server. So always consider the application when adding additional processors to solve a processor bottleneck. Ask the software vendor if the application is designed to take advantage of Hyper-Threading or greater SMP capability. Also, look at the System: Processor Queue Length counter to see if the application is spawning a large number of threads, which makes it a candidate for solutions with Hyper-Threading or a larger number of processors.<br />
<br />
19.2.1 Finding CPU bottlenecks You can use System Monitor processor object counters to help you determine if the processor is the bottleneck by creating a chart as discussed in 14.1.3, “Using System Monitor” on page 481. After gathering performance data, you should analyze it according to the recommendations in Table 19-1 on page 658. Also consider that Windows Server 2003 has a much more efficient scheduling routine than the scheduler that is used by Windows 2000. Windows 2000 attempted to balance thread execution across all processors, while Windows Server 2003 attempts to keep threads that are associated with a particular process together by executing on a home or preferred processor. This affinity assignment is done to enhance NUMA-based systems (see 7.2.3, “NUMA” on page 101), and it greatly reduces processor-cache-to-cache migration of thread code and data that previously occurred in Windows 2000. Affinity assignment significantly increases system efficiency at the expense of a potentially less balanced execution load. So, under some workloads, expect to<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
657<br />
<br />
see more unbalanced CPU utilization with Windows Server 2003, which is more efficient and results in higher overall performance. Examine the indications of processor bottlenecks based on the object counter readings that you have obtained, then perform the recommended actions in Table 19-1 to rectify the situation. Tip: In Windows Server 2003, you can set processor counters to monitor a specific processor individually or total processor usage of the server. Always examine both sets of counters to determine if one or more processors is causing a bottleneck that you do not see by looking at the %Total Processor Utilization counter. Table 19-1 Performance console counters for detecting CPU bottlenecks<br />
<br />
Counter<br />
<br />
Description<br />
<br />
Processor: %Processor Time<br />
<br />
This counter is the percentage of time the processor is busy. When this counter is consistently over 75% to 80%, the processor has become a system bottleneck. You should examine processor utilization for each individual CPU (instance) as well as the average for all CPUs in your server. From this counter, you can tell if one or a few CPUs are being used significantly more than the others. This is often indicative of an affinitized application which has out-grown its allocated processing resources.<br />
<br />
Processor: %Privileged Time<br />
<br />
This counter measures the time the processor spends performing operating system functions and services. In general, most server applications spend about 20% to 50% of the time in privileged time. If you spot excessive privileged time you will need to determine if the application is making excessive kernel calls or a device driver is operating incorrectly.<br />
<br />
Processor: %User Time<br />
<br />
This counter is the percentage of processor time spent in user mode executing the server application. The percentage of time spent in %User Time compared to %Privileged Time will help you identify whether the application or the OS and device drivers are likely causing the CPU bottleneck. If the %User Time makes up the overwhelming component of %Processor Time, the application is consuming most of the processing cycles. In general, this is ideal because we want the application to execute as much of the time as possible. But this also means the application has to be reconfigured or modified in some way for the system to be made more efficient. For verification, you might have to examine CPU usage by process to identify which application process is using the majority of the processing time.<br />
<br />
658<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Counter<br />
<br />
Description<br />
<br />
System: Processor Queue Length<br />
<br />
This counter is the instantaneous length of the processor queue in units of threads. All processors use a single queue in which threads wait for processor cycles. With Hyper-Threading processors, a sustained processor queue length that is higher than 10 threads per processor might indicate congestion, which indicates a processor bottleneck. This bottleneck means that the processor cannot handle the concurrent thread execution requirements. The good news is that a high System Processor Queue Length indicates the application will scale to a higher performance level on a system configured with additional processors. When the System Processor Queue Length is low, the application might not scale on a system with a greater number of processors unless some configuration parameter is limiting the number of concurrent threads generated by the application. Some applications increase the thread depth when they detect additional processors. So it is hard to know with certainty that an application will not take advantage of a greater number of processors. Confirm this with your software vendor.<br />
<br />
Figure 19-1 shows a sample Performance console chart setting for detecting processor bottlenecks.<br />
<br />
Figure 19-1 Chart setting for finding processor bottlenecks<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
659<br />
<br />
19.2.2 Processor subsystem performance tuning options If you have determined that your processor subsystem is unhealthy and causing performance problems you have several basic choices, these include: Upgrade to a faster processor Add processors to the system (if possible) Optimize software causing the bottleneck Now. let us review each of the choices and determine which strategy makes sense: Upgrade to faster processors or processors with larger caches Upgrading to faster processors is usually the safest way to solve a processor bottleneck because you can be assured the application will execute faster. Adding processors places increased threading requirements on the software. However, adding faster processors executes the existing threads faster without any additional software support. See 4.4, “Rules of thumb” on page 76 for approximate performance gains from faster processors and larger processors caches to level set your expectations. Remember processors that have higher frequency usually have other micro-architecture features that effect performance. Do not expect processor performance to match exactly frequency improvements. For example, a 3.0 GHz processor is not twice as fast as a 1.5 GHz processor. Add processors Only add multiple processors to improve performance when you are certain the applications have proper threading and can take advantage of the additional processors. Do not take this point lightly. Not all applications scale and can take advantage of additional processors. In general, the majority of older server applications work well with two processors. Many, not most, run well on four-way SMP systems. And in general, the applications that can take advantage of eight or more processors are typically enterprise middleware and database applications. One key point to remember: If the current system is not constantly running at very high CPU utilization—that is, greater than 80% to 90%—then adding processors will likely not improve system level performance. If the CPU subsystem is not saturated, then adding more processors or faster processors will simply reduce total sustained CPU utilization with a slight increase in performance. In addiction, adding additional processors always introduces some additional overhead (for example, greater scheduling and bus contention overhead).<br />
<br />
660<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Optimize the software environment If %Kernel Time is high, the operating system or device driver is using most of the processing time. This fact could be because the application simply makes many calls to the operating system or because of an inefficient device driver or improper operating system configuration. If the majority of time is spent in %User mode, then the applications are using most of the processing power. In this case, examine whether you can configure the application to be more efficient. Also, examine whether you can schedule some CPU-intensive jobs to run during off-peak hours. You can use the AT command that is included in Windows Server 2003. This command is useful especially when doing system backups.<br />
<br />
19.3 Analyzing memory bottlenecks As mentioned in Chapter 11, “Microsoft Windows Server” on page 295, Windows Server 2003 has the ability to self-optimize but only for certain aspects of the operating system. This optimization process focuses on memory caching and virtual memory management. The Windows memory manager adjusts the amount of caching memory that is available to best suit operating conditions. However, memory is one of the most common sources for server bottlenecks, so do not overlook this section. When configured for file serving mode, Windows Server 2003 favors a large disk cache where most of the available memory is assigned to the disk cache, and only some memory is available for loading programs. In application server mode, the operating system reduces the amount of memory available to the disk cache and maximizes the memory available to run applications. See 11.6, “File system cache” on page 309 for details. The amount of memory that the operating system and applications have available will significantly influence your server's performance. You can find more information about the memory subsystem in Chapter 8, “Memory subsystem” on page 129.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
661<br />
<br />
19.3.1 Paged and non-paged RAM Physical memory in Windows Server 2003 is divided into paged and non-paged areas, as shown in Figure 19-2. Non-paged memory is critical memory used for drivers, the kernel, or application contents that must remain in RAM and is never paged to the page file device because the non-paged memory is needed for general operation of the system. If it were paged out to disk, the operating system might not have all the data it needs to access the page file device, which could create a dead lock and the system might crash. To avoid this potential for a crash, the operating system marks critical memory as non-paged, thereby pinning these memory locations into physical memory. Paged memory is non-critical memory buffers that can be written out to the page file on disk because the contents are not required for general execution of the operating system. Non-Paged Pool<br />
<br />
User<br />
<br />
Paged Pool<br />
<br />
Application working set<br />
<br />
Disk cache Kernel<br />
<br />
User<br />
<br />
Application working set<br />
<br />
User<br />
<br />
Application working set<br />
<br />
(Dynamically Adjusted)<br />
<br />
Page file Figure 19-2 Windows 2000 memory definition<br />
<br />
Programs can have a portion of their memory space set as non-pageable, but the majority of program functions are pageable. At program load, Windows Server 2003 loads all code that is needed for execution. However, usually much of the code and data is not in constant use as the program executes. As a result, the operating system marks the unused memory storing data to be sent into the page file and marks the rarely used memory storing code for deletion. Code is never paged, because it can simply be read back from disk—only data is written to the page file. However, these unused objects in memory are not automatically paged out to the page file on disk. The actual paging to disk occurs only when other programs (or the disk cache) require additional memory and no free memory is available. When this occurs, the operating system frees memory used to store code that<br />
<br />
662<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
has been marked as unused, and writes the data that has been marked as unused to the pagefile. This frees up memory which in turn is allocated to the application making the request for more memory. When this memory paging occurs we call this memory pressure. Memory pressure occurs when applications are constantly requesting memory buffers that are not freely available in the system, and the operating system is forced to perform housekeeping to delete the least recently used code memory, and swap the least recently used data to the swap device. Clearly, when memory pressure occurs, system performance suffers. So if we can detect memory pressure (physical swapping), we can detect the slow-down due to insufficient memory.<br />
<br />
19.3.2 Virtual memory system Because almost all of the System Monitor performance counters for memory relate to virtual memory, they cannot be used to directly diagnose physical memory pressure. However, there are several counters that you can use to indicate that the lack of sufficient memory is causing a performance problem. These counters are:<br />
<br />
Memory: Available MBytes This counter is the amount of physical memory that is available to processes that are running on the server (in MB).<br />
<br />
Memory: Page Reads/Sec Memory: Page Writes/Sec These two counters indicate hard paging to disk is occurring. Nearly every server has pages per second counts that occur during normal operation, because page misses occur and the resulting miss can be serviced from a memory page that has not yet been paged out to disk. Remember, unused pages are marked for swapping out to the page file but will remain in system memory as long as available physical memory is not constrained and consequently memory pressure does not exist. So it would be incorrect to use Pages/Sec as an indicator to determine if physical memory capacity is insufficient. Only when paged data is actually written or read from the page file on disk is the server experiencing a performance bottleneck from lack of memory.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
663<br />
<br />
You can use the Performance console memory object counters listed in Table 19-2 to help you determine memory bottlenecks. Table 19-2 Performance console counters for detecting memory bottlenecks<br />
<br />
Counter<br />
<br />
Description<br />
<br />
Memory: Page Reads/sec<br />
<br />
This counter is the number of disk read operations performed for physical paging. Generally, if this value has periods of sustained value of over 50-100 operations per second it indicates significant paging activity and insufficient memory capacity might be causing slow performance.<br />
<br />
Memory: Page Writes/sec<br />
<br />
This counter is the number of disk write operations performed for physical paging. Generally, if this value has periods of sustained value of over 200-300, it indicates a great deal of paging activity. Memory capacity might be the bottleneck in your system.<br />
<br />
Memory: Available MBytes<br />
<br />
In Windows Server 2003, this counter indicates the amount of remaining physical memory in MB available to applications. If the server is configured to be optimized for file serving applications, this counter will normally be low, as Disk Cache Manager uses extra memory for caching and then returns it when requests for memory occur. If this value stays at less than 20% to 25% of installed RAM, it is an indication that you do not have enough virtual memory. If this counter is consistently low after running an application, this indicates a memory leak. Check the application causing a memory leak.<br />
<br />
Memory: Pool Nonpaged Bytes<br />
<br />
This counter indicates the amount of RAM in the non-paged pool system memory area where space is acquired by operating system components as they accomplish their tasks. If this value has a steady increase without a corresponding increase in activity on the server, it might indicate that a process that is running has a memory leak, and it should be monitored closely.<br />
<br />
19.3.3 Performance tuning options Typically, you have two choices for solving memory bottlenecks. These options are, in order of importance: Increase the total memory capacity of the server If paging is unavoidable, move the page file to a faster disk or array<br />
<br />
Adding memory Many performance problems are solved by simply increasing memory capacity of the server. However, you need to consider carefully the memory object counters, the system hardware, and application configuration to ensure maximum performance gains. There is always a point at which adding memory capacity will not help increase throughput. In general, for most servers running a single application, this limit is<br />
<br />
664<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
at about 3 GB to 4 GB of memory, or whenever the application has no need for additional memory. Also, latent bottlenecks in the network subsystem or processors will put a ceiling on the maximum gains obtained by adding memory. Figure 19-3 shows the effect of adding and removing memory to and from a file server. Note: When adding DIMMs, ensure the server has DIMMs populated in all the slots needed to utilize the maximum interleave and concurrency the memory subsystem offers. See 8.1.9, “Memory interleaving” on page 143for additional details.<br />
<br />
Server with additional memory<br />
<br />
Throughput or transactions per second<br />
<br />
Server with insufficient memory<br />
<br />
Increasing number of users or server workload Figure 19-3 Effect of memory changes in a file server<br />
<br />
Adding memory affects the performance of the server in the following ways: Adding memory to a server improves the disk cache hit rate or reduces system paging, which increases the sustained server throughput rate (assuming that the caching algorithms that are used are effective for the particular application). This reduces the disk I/O rate and increases network utilization because the server is now able to respond to requests at a faster rate. As a result, a slow network adapter can end up becoming the next bottleneck after memory is added to address the memory bottleneck. Higher disk cache hit rates or lower paging also translate into higher CPU utilization, again because the processors are no longer waiting as much for disk I/O to complete. So, poor CPU head-room can also reduce the potential performance gains from adding memory.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
665<br />
<br />
These points emphasize the importance of configuring a server properly for balanced performance.<br />
<br />
Unavoidable paging The optimal situation for Windows servers is to have low or no sustained paging. Because Windows needs a page file, there will usually be some paging at initial start-up of the server. Paging activity often happens as an application warms-up and in most cases should be minimal afterwards. In some cases, however, even with sufficient memory capacity, excessive paging cannot be eliminated. For example, the Windows operating system provides a facility to support data sharing between applications which uses the paging feature of Windows. This facility is called memory mapped files. Memory mapped files enable applications to share large data items that cannot fit in physical memory by using the page file system as a virtual large shared memory buffer. In this case, one process can store data quickly into the page file and another process can access quickly that data without the overhead of using the file system to share data on disk. For more information about memory mapped files see the MSDN® Web site: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dngenl ib/html/msdn_manamemo.asp However, the bottom line is that when applications are using memory mapped files, significant paging occurs and no amount of additional memory reduces the memory mapped component of paging to disk. Our experience is that only a few applications use memory mapped files. So in general, if the server has 3 GB or 4 GB of memory and is only running a single application that does not use 4 GB+ memory-access techniques (such as Windows AWE), and it still pages, then it is probably using memory mapped files. In such instances, our recommendation is to put the page file on the fastest disk array possible, such as a local RAID-1 array.<br />
<br />
Recommendations So our recommendations for solving memory capacity issues are: Check Page Reads/sec and Page Writes/sec. If these counters are greater than 50-100 pages/sec, and the server is configured with less than 3 GB to 4 GB, add memory up to the 3 GB to 4 GB limit per supported application. If after the memory capacity is increased to 4 GB per supported application, the server still pages significantly (greater than 200-300 pages/sec) then<br />
<br />
666<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
assume the application is using memory mapped files and proceed to move the page file to a fast RAID device. Also check the Memory Available MB counter to see if the available memory is low. This is an indicator that the server is running low on physical memory. Again add memory up to the 4 GB limit for applications, and recheck paging. In summary, where memory capacity is at the maximum supported by the application remember a page file competes with other disk accesses to that same disk. And after sufficient memory is added there is no way to reduce paging by adding additional memory. So, put the paging file on a low activity disk, or better yet, spread it over a dedicated array of disks for paging only. For best performance do not use IDE or EIDE disks for this paging function. They do not support asynchronous I/O, which means only one access is supported at a time. You can also schedule memory-intensive applications during off-peak hours. You can use the AT scheduler that ships with the operating system. For example, it does not make sense to do a tape backup during heavy system utilization. Note: You need to also read the suggestions in 8.11, “Memory rules of thumb” on page 165.<br />
<br />
19.4 Disk bottlenecks Windows Server 2003 retrieves programs and data from the disk. The disk subsystem can be the most important aspect of server performance, but problems can be hidden by other factors, such as lack of memory, so always check memory capacity before modifying the disk configuration. Disk subsystem operation is discussed in Chapter 9, “Disk subsystem” on page 169. You need to review this chapter before analyzing disk bottlenecks. Performance console disk counters are available with either the LogicalDisk or PhysicalDisk objects: LogicalDisk monitors the operating system partitions stored on physical drives or physical arrays. After identifying a physical disk bottleneck, if multiple logical partitions are on the physical array, this object is useful to determine which partition is causing the disk activity, possibly indicating the application or service that is generating the requests.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
667<br />
<br />
PhysicalDisk counters reflect the activity to the hard disk drives, or arrays of hard disk drives, and is useful for monitoring all disk activity to the drives. Our initial analysis is always performed using physical disk counters because we first want to identify if the system has a hardware problem. If after determining the physical disk configuration is not optimal, review the logical disk counters to determine if disk IO to one or more logical drives on the busy physical drive can be moved to another physical disk or array with less I/O traffic. Note: For initial analysis of disk performance bottlenecks, always use physical disk counters.<br />
<br />
Tip: Activating disk performance counters In Windows 2000, physical disk counters are enabled by default, but logical disk counters are disabled by default. If you use software RAID, you need to enable logical disk counters using the command: DISKPERF -YV Keeping these settings on all the time draws about 2% to 3% CPU but if your CPU is not a bottleneck, this is irrelevant and can be ignored. Type DISKPERF -? for more help on the DISKPERF command, and type just DISKPERF (no parameters) to get the current status.<br />
<br />
668<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
19.4.1 Analyzing disk bottlenecks You can use the physical disk object counters listed in Table 19-3 to help you determine if you have disk bottlenecks. Then examine the indications of disk bottlenecks based on the object counter readings. Afterwards, you should perform appropriate actions to respond to the situation. Table 19-3 Performance counters for detecting disk bottlenecks<br />
<br />
Counter<br />
<br />
Description<br />
<br />
Physical Disk: Avg. Disk sec/Transfer<br />
<br />
This counter is a key counter that indicates the health of the disk subsystem. This counter is the time to complete a disk I/O operation. For optimal performance, this should be less then 20-25 ms for non-clustered systems, and no higher than 40-50 ms for clustered disk configurations. In general, this counter can grow to be very high when insufficient numbers of disks, slow disks, poor physical disk layout, or severe disk fragmentation occurs.<br />
<br />
Physical Disk: Avg. Disk Queue Length<br />
<br />
This counter is the average number of both read and write requests queued to the selected disk during the sample interval. If this value is consistently over 2-3 times the number of disks in the array (for example, 8-12 for a 4-disk array), it indicates that the application is waiting too long for disk I/O operations to complete. To confirm this assumption, always check the Avg. Disk Sec/Transfer counter. Also, the Avg. Disk Queue Length counter is a key counter for determining if a disk bottleneck can be alleviated by adding disks to the array. Remember, adding disks to an array only results in increased throughput when the application can issue enough multiple requests to the array to keep all disks in the array busy. For optimal disk performance, we want the Avg. Disk Queue Length to be no more than 2 or 3 times the number of physical disks in the array. Also, in most cases the application has no knowledge of how many disks are in an array because this information is hidden from the application by the disk array controller. So unless an application configuration parameter is available to adjust the number of outstanding I/O commands, an application will simply issue as many disk I/Os as it needs to accomplish its work, up to the limit supported by the application or disk device driver. Before adding disks to an array to improve performance, always check the Avg. Disk Queue Length counter and only add enough disks to satisfy the 2-3 disk I/Os per physical disk rule. For example if the array shows an Avg. Disk Queue Length of 30 then an array of at most 10-15 disks should be used.<br />
<br />
Physical Disk: Avg. Disk Bytes/Transfer<br />
<br />
This is the average number of bytes transferred to or from the disk during write or read operations. This counter can be used as an indicator of the stripe size that should be used for optimal performance. For example, always create disk arrays with a stripe size that is at least as large as the average disk bytes per transfer counter value as measured over an extended period of time.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
669<br />
<br />
Note: Never use the %Disk Time physical disk counter to diagnose server bottlenecks. This counter is the percentage of elapsed time that the selected disk drive is busy servicing read or write requests. However, this counter is only useful with IDE drives, which, unlike SCSI disks, can only perform one I/O operation at a time. The %Disk Time counter is derived by assuming the disk is 100% busy when it is processing an I/O and 0% busy when it is not. The counter is a running average of the 100% versus 0% count (binary). SCSI array controllers can perform many hundreds or thousands of I/Os per second before they encounter bottlenecks. Most array controllers can perform two to three disk I/Os per drive before a bottleneck occurs. For example, if an array controller with 60 drives has one disk I/O to perform at all times it will be 100% utilized according to the % Disk Time counter. However, that array could actually be issued 120-180 I/Os before a true bottleneck occurs.<br />
<br />
19.4.2 Performance tuning options After verifying that the disk subsystem is a bottleneck, a number of solutions are possible. These solutions include: Verify stripe size is at least as great as the sustained Avg. Bytes/transfer counter value for each array. If not, the array could be doing multiple physical disk I/Os to satisfy each request. Offload files that are experiencing heavy I/O processing to another server or to another array on the same server. Add more RAM. Use faster speed disks. Add more disk drives to an array in a RAID environment. This spreads the data across multiple physical disks and yields increased I/O rates.<br />
<br />
670<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 19-4 shows the effect of putting a faster disk subsystem on a file server.<br />
<br />
Throughput or transactions per second<br />
<br />
Server with faster disk subsystem<br />
<br />
Server with slower disk subsystem<br />
<br />
Increasing number of users or server workload Figure 19-4 Effect of adding a faster disk subsystem to the file server<br />
<br />
This is a good time to review how the disk subsystem's performance affects the overall performance of the server. A faster disk subsystem usually improves sustained transaction rate for the server at the lower part on the right of the curve in Figure 19-4. The peak of the curve is the peak sustainable throughput of the network adapter, and the lower part of the curve to the right represents the sustainable throughput of the disk subsystem. The disk subsystem might only slightly affect performance under light loads because most requests are serviced directly from the disk cache. In this case, network transfer time is a relatively large component and disk transfer times are hidden by a high frequency of disk cache accesses. As the server disk performance improves, increased network adapter and CPU performance is required to support greater disk I/O transaction rates. Adding memory will increase system memory disk cache, which in effect reduces disk I/O traffic, thereby improving server throughput and response times. Adding memory should be the first course of action before reconfiguring the disk subsystem. Again, review 19.3, “Analyzing memory bottlenecks” on page 661 first to make certain memory capacity is optimal before acting on a disk bottleneck.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
671<br />
<br />
When disk bottlenecks are detected, one option is to replace the slow disk with a faster one. However, consider that faster disks usually result in system level improvements on the order of 20% to 40%, not 2 or 3 times. So if the performance problem is a mild one, using faster disks can be considered. But in general, where performance improvements must be significant, adding disks to the array is usually the best choice. Adding disks is the safe way to improve performance and usually the most cost effective because one does not have to replace current hardware. However, always determine the concurrent I/O demand of your server application before adding disks. Checking the Avg. Disk Queue length will help you understand how many disks to add to the array. This can be calculated simply by dividing the Avg. Disk Queue length counter by 2 for very best performance or 3 for best price-performance. So, for example, if the sustained Avg. Disk Queue length is 12, then configure the array with at most 6 disks (for optimal performance) and 4 disks for best price performance.<br />
<br />
19.5 Network bottlenecks Network performance trouble-shooting can be a very complex task. This complication is in part because the performance counters obtained by System Monitor only represent traffic flow to and from the server for which the counters were monitored. The performance counters do not reflect the total traffic in the network. So, while the counters for a particular server might seem reasonable, the network itself could be experiencing heavy load and be causing slow server response times as seen by the users. In general, when poor network performance is suspected, you must rely on help from an experienced networking professional. The complete debug of network performance problems is complicated and beyond the scope of this redbook. However, there are a few performance counters that can be used to perform a basic diagnosis of server network adapter bottlenecks and help you know when to call in the experts.<br />
<br />
672<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
19.5.1 Finding network bottlenecks The network performance object counters that should be investigated are presented in Table 19-4. You should examine the indications of network bottlenecks based on the object counter readings. Also included here are suggestions that could alleviate the situation. Table 19-4 Performance console counters for detecting network bottlenecks<br />
<br />
Counter<br />
<br />
Description<br />
<br />
Network Interface: Bytes Total/sec<br />
<br />
Sustained values over 50% to 60% of the network adapter’s available bandwidth are cause for concern. Expected maximum sustained throughput for a Gigabit Ethernet in a Xeon or Xeon DP processor-based server supporting general commercial workloads with a 70-30 send-receive traffic mix is about 1500 Mbps. To be conservative, detailed network analysis is warranted if the Bytes Total/sec value is over about 700-800 Mbps. Maximum sustained full duplex throughput for 100 Mbps Ethernet is about 150 Mbps.<br />
<br />
Network Interface: Bytes Received/sec<br />
<br />
This counter is a network subsystem primary counter. Sustained values over 50% to 60% of maximum sustained throughput in the receive direction should be investigated by a network administrator to determine if the network is a bottleneck. Most Gigabit Ethernet adapters can sustain about 800 Mbps in the receive direction. Maximum sustained throughput for 100 Mbps Ethernet in the receive direction is about 80 Mbps.<br />
<br />
Network Interface: Bytes Sent/sec<br />
<br />
Sustained values over 50% to 60% of maximum sustained throughput in the send direction should be investigated by a network administrator to determine if the network is a bottleneck. Most Gigabit Ethernet adapters can sustain about 800 Mbps in the send direction. Maximum sustained throughput for 100 Mbps Ethernet in the send direction is about 80 Mbps.<br />
<br />
Network Interface: Packets/sec and Network Interface: Packets Sent/sec and Network Interface: Packets Received/sec<br />
<br />
Packets/sec rates should be no higher than about 50% to 60% of maximum packet/sec rates listed in Table 19-5 on page 676 and Table 19-6 on page 677. Use the Bytes/Total/sec counter value divided by Total Packet/sec counter value to calculate average Bytes per Packet or average packet size for your server workload. Then using the calculated average packet size (bytes/packet), use Table 19-5 on page 676 and Table 19-6 on page 677 to determine the maximum packet/sec rate sustainable for the adapter speed being used. We recommend detailed network analysis If the sustained values are 50% or greater than the values listed in the table.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
673<br />
<br />
19.5.2 Analyzing network counters A network adapter can have two primary types of performance bottlenecks: Data rate bottleneck (saturating the network interface) Packets per second bottleneck (saturating the adapter’s processor) The first type of bottleneck occurs when the network adapter is running at the maximum sustainable data rate. Data rate bottlenecks are the first bottleneck most people think of when diagnosing server network performance bottlenecks. However, the truth is, bandwidth bottlenecks rarely occur. In general a packet per second rate bottleneck, or the number of packets per second that the network adapter can process before dropping packets, is the most likely network bottleneck to occur in servers. This bottleneck is because many server applications communicate by rapidly sending small messages—about 64-512 bytes in size. These smaller data packets do not saturate the data bandwidth of the network, but they can often saturate the ability of the LAN adapter to process send and receive packets. Therefore, to diagnose server LAN adapter bottlenecks, we need a method to identify both bandwidth bottlenecks and packet rate bottlenecks. Diagnosing bandwidth bottlenecks is not as straight forward as you might think. In general, we want to determine if the LAN adapter is moving data at a rate which approaches the maximum sustainable rate of the network adapter being used. Keep in mind, however, that the server being analyzed is most likely not the only server in the network, so when examining data bandwidth counters, remember that performance counters only reflect the data rate to and from the monitored server, not the total traffic of the network. To use System Monitor performance counters as an indicator for network bottlenecks, you should adopt a conservative approach. When the server is sustaining more than about 50% of the available network bandwidth you should examine the network as a potential bottleneck. Tip: If your sustained network bandwidth is 50% or greater than the potential bandwidth of your server, engage a network expert to help you diagnose the bottleneck. Of course if you know the server is the only server in the network, and there is little peer-to-peer traffic, then use a less conservative value of up to 75% to 80% of sustainable throughput to indicate a network bottleneck. Because Ethernet is full duplex, it is possible for a 100 Mbps Ethernet controller to have a theoretical data rate of 200 Mbps (100 Mbps in each direction) because<br />
<br />
674<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
the network adapter can transmit 100 Mbps and receive 100 Mbps concurrently. In practice, these theoretical rates are not sustainable, and you should expect the best case sustained data rate in each direction to be about 80 Mbps. This discussion also brings up an important point regarding network bottleneck detection. Because Ethernet is full-duplex it is possible to have either a transmit bottleneck, a receive bottleneck, or both. So our methodology must check both the send, and receive throughput rates to determine the presence of any network adapter bottlenecks. For Gigabit Ethernet, the maximum sustained data rates are about 10 times the 100 Mbps rates and one can expect about 750-800 Mbps in each direction. Note that this assumes you have a system capable of driving the TCP/IP stack to that level of bandwidth. Generally this requires a modern Xeon MP or Xeon DP-based server with two or more processors and a front-side bus of at least 400 MHz. See 8.2, ”Factors affecting network adapter performance” for an analysis of the effects of these server components on network throughput. For this discussion, we assume that other subsystems in your server are not the bottleneck, and we use the 80 Mbps and 800 Mbps throughput limits for 100 Mbps and 1 Gbps Ethernet controllers, respectively, as our maximum sustainable unidirection throughput values. Keep in mind, we have to keep our thresholds conservative because we do not know the total external traffic in the network, nor are all Ethernet adapters capable of these maximum data rates. So in general, we should start to investigate our network in more detail whenever the sustained bandwidth is equal to half the total sustainable bandwidth in any direction of the network. So for 100 Mbps Ethernet, start to get concerned when you spot about 40 Mbps of sustained bandwidth, in any direction, transmit or receive. In addition, for Gigabit Ethernet start, call in the network experts whenever you spot over 400 Mbps data rates for either send or receive bytes/sec counters. That covers network bandwidth thresholds, but what about packet/sec rate thresholds? Because many applications communicate using small packets of data, it is entirely possible for the server to have a packet per second rate bottleneck long before it reaches the maximum bandwidth of the network. It is difficult to offer specific advice without detailed knowledge of the server configuration and particular network adapter. However, because recent server Ethernet adapters have matured such that they have similar performance, you can employ tables of measured packet/sec rates to help identify when you need to be concerned that your server might be experiencing a packet/sec rate bottleneck.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
675<br />
<br />
The following tables show throughput in packets/sec for 100 Mbps Ethernet (Table 19-5) and Gigabit Ethernet (Table 19-6 on page 677). These are reasonable peak rates that you can expect from leading server network adapters. Expect much lower throughput if you use less expensive client adapters in your server. Table 19-5 100 Mbps Ethernet: Expected maximum (peak) packet rates for given packet sizes<br />
<br />
Average packet size (bytes)<br />
<br />
100% server receives (packets/sec)<br />
<br />
70/30 send/receive (packets/sec)<br />
<br />
100% server sends (packets/sec)<br />
<br />
64<br />
<br />
105,790<br />
<br />
103,014<br />
<br />
126,553<br />
<br />
128<br />
<br />
89,939<br />
<br />
90,661<br />
<br />
90,428<br />
<br />
256<br />
<br />
55,897<br />
<br />
67,044<br />
<br />
56,140<br />
<br />
512<br />
<br />
31,789<br />
<br />
39,994<br />
<br />
31,788<br />
<br />
1024<br />
<br />
17,028<br />
<br />
21,701<br />
<br />
17,024<br />
<br />
1460<br />
<br />
12,206<br />
<br />
15,610<br />
<br />
12,202<br />
<br />
2 KB<br />
<br />
17,021<br />
<br />
23,871<br />
<br />
17,024<br />
<br />
4 KB<br />
<br />
12,998<br />
<br />
17,766<br />
<br />
13,001<br />
<br />
8 KB<br />
<br />
12,999<br />
<br />
18,278<br />
<br />
13,002<br />
<br />
16 KB<br />
<br />
12,999<br />
<br />
18,238<br />
<br />
13,004<br />
<br />
676<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Table 19-6 Gigabit Ethernet: Expected maximum (peak) packet rates for given packet sizes<br />
<br />
Average packet size (bytes)<br />
<br />
100% server receives (packets/sec)<br />
<br />
70/30 send/receive (packets/sec)<br />
<br />
100% server sends (packets/sec)<br />
<br />
64<br />
<br />
125,780<br />
<br />
169,218<br />
<br />
146,017<br />
<br />
128<br />
<br />
164,597<br />
<br />
161,256<br />
<br />
140,588<br />
<br />
256<br />
<br />
153,649<br />
<br />
158,360<br />
<br />
137,616<br />
<br />
512<br />
<br />
135,439<br />
<br />
146,796<br />
<br />
151,588<br />
<br />
1024<br />
<br />
114,985<br />
<br />
120,586<br />
<br />
158,479<br />
<br />
1460<br />
<br />
107,760<br />
<br />
114,046<br />
<br />
120,701<br />
<br />
2 KB<br />
<br />
122,901<br />
<br />
157,194<br />
<br />
165,803<br />
<br />
4 KB<br />
<br />
124,334<br />
<br />
151,746<br />
<br />
123,718<br />
<br />
8 KB<br />
<br />
102,238<br />
<br />
157,377<br />
<br />
126,362<br />
<br />
16 KB<br />
<br />
115,153<br />
<br />
158,683<br />
<br />
98,024<br />
<br />
Determine if you have a full-duplex packet rate bottleneck. You can use these tables to detect full-duplex packet rate bottlenecks in the following way: 1. Divide the Packets/sec counter value by the Bytes Total/sec counter value to derive the average full-duplex packet size for your server. This value is the average packet size for both send and receive traffic. 2. Compare your calculated average packet size to the corresponding table above. The tables have three columns, one for 100% receive, 70-30 receive-send, and 100% send. Use the 70-30 packet rate column because it is a reasonable indicator of full-duplex packet rates for a typical business application. 3. Select the packet size which is closest to your calculated value, then read that row across to arrive at the maximum send/receive packet/sec rate one could expect from the network adapter installed in your server. 4. Compare this to the System Monitor counter Packets/sec that you measured for your server. Unless you know your server is on an isolated network, you should be concerned and call in networking expertise if the Packet/sec rate counter value is close to 50% of the maximum values supplied in the table. If the server is on an isolated network than expect to achieve up to 70% or greater of the listed Packet/sec rate before network bottlenecks usually occur.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
677<br />
<br />
Now check for both send and receive packet rate bottlenecks: 1. Perform the same calculation for sent packets by dividing the Send Bytes/sec counter value by the Packets Sent/sec counter, match up your calculated average send packet size to the closet value in the table and read across to the send column to identify the peak value for the network speed you are using (100 Mbps or Gigabit). 2. Compare this peak value to the Packet Sent/sec counter value from System Monitor for your server. Remember, you should be concerned when the send packet rate that you measured is within 50% of the value that is listed in the table for the corresponding network speed that you are using. 3. Repeat this process for the receive packet rate. Perform the same table look-up using the calculation for receive packet rates by dividing Receive Bytes/sec by Packets Receive/sec counters. Use this value to index into the table to the closest packet size to match your calculated average, then compare the peak Receive packet/sec rates listed in the table to the measured Receive Packet/sec rate to detect receive side packet rate bottlenecks. Again you should be concerned and call in networking expertise if your calculated packet/sec rate is close to 50% of the maximum values supplied in the table.<br />
<br />
19.5.3 Solving network bottlenecks After verifying that the network subsystem is the bottleneck, a number of solutions are possible. Here are some actions that you can take to respond to network bottlenecks. However, in any event, always consult a network expert before making modifications to your system. Because any modifications must be done only after considering the network traffic not seen by the System Monitor counters.<br />
<br />
Hardware approach Consider the following items: Use 64-bit PCI-X LAN adapters if available. These adapters are faster, more efficient, and as PCI bus master devices, do not consume significant amounts of CPU time. A good LAN adapter will incorporate features to reduce contention in the network as well as provide good performance with minimum interrupt generation.<br />
<br />
678<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
You can also move to a higher-speed LAN adapter such as Gigabit Ethernet. Figure 19-5 shows the effect of adding a faster network adapter to the file server.<br />
<br />
Throughput or transactions per second<br />
<br />
Adding a faster network adapter benefit is maximized during high cache hit rate Benefit drops off as disk I/O rate becomes the bottleneck<br />
<br />
Increasing number of users or server workload<br />
<br />
Figure 19-5 Effect of adding faster network adapter to the file server<br />
<br />
The network adapter’s speed impacts the overall performance of the file server in the following ways: – It improves the maximum peak transaction rate of the server. – It might only slightly affect performance under heavy loads because the server is disk bound as it waits for seeks. In this case, the total network transfer time is a relatively small component of the overall transaction. – As network adapter performance improves, increased CPU performance is required to service the increased request rates from the users on the LAN. Balance the network load using segmentation or by having more than one LAN adapter in the server. Alternatively, set workgroup switches to isolate traffic to appropriate segments. Switches serve no useful purpose if most traffic has to travel from one port to another on the same segment. The secret of using switches is to keep local traffic within a segment and stop it from interfering with another segment. Create multiple networks or subnetworks. This helps in handling unnecessary broadcasts over the network. Upgrade to better performing routers and bridges. Add more servers to the network. In this way, you distribute the processing load to other servers.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
679<br />
<br />
Software tuning options Consider the following items: Use recent LAN adapter device drivers. LAN adapter manufacturers usually develop their latest drivers to address bugs and inefficiencies in previous releases. The driver should fully support the latest Windows networking revisions. The tool INTFILTR which is an interrupt affinity tool that allows you to bind device interrupts to specific processors on SMP servers. This is a useful technique for maximizing performance, scaling, and partitioning of large servers. It can provide up to a 20% network performance increase. For more information, go to one of the following: Windows 2000 Server: http://support.microsoft.com/?kbid=252867 Windows Server 2003: http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69-57 ff-4ae7-96ee-b18c4790cffd&displaylang=en You need to remove network services that are not being used to free up the system’s needed resources. Follow these steps: a. b. c. d.<br />
<br />
680<br />
<br />
Click Control Panel → Network and Dial-up Connections. Select any connection, and then select File → Properties from the menu. Choose a service to be removed, and then click Uninstall. Confirm Yes, and then Close. Figure 19-6 shows you how to remove network services from the Windows 2000 server.<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Select the service that you want to remove, and then click Uninstall.<br />
<br />
Figure 19-6 Removing network services from Windows 2000 Server<br />
<br />
You need to perform additional steps if you want to remove other networking services from the Windows 2000 server than those listed in Network and Dial-up Connections: a. Click Control Panel → Add/Remove Programs → Add/Remove Windows Components. b. Select Networking Services or Other Network File and Print Services. c. Click Details... and then deselect services that you do not need. d. Click OK, then Next, and Finish. Figure 19-7 shows you how to remove additional network services from the Windows 2000 server.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
681<br />
<br />
Deselect services that you want to remove, then click OK.<br />
<br />
Figure 19-7 Removing unnecessary Windows 2000 networking services<br />
<br />
Network design issues Consider the following items: Optimize the protocol binding order. The system looks at the first loaded protocol to fulfill any network I/O requests. You need to load the most frequently used protocol first to get the best performance. You can gain performance benefits by optimizing the binding order. For example, if you have a relatively small network, NetBEUI is an appropriate protocol that performs faster. If you need to communicate with remote servers through TCP/IP on a less frequent basis, it would be better to configure your protocol binding order to make NetBEUI first. To change the binding order of a protocol, follow these steps: a. Click Control Panel → Network and Dial-up Connections. b. Select any connection, and then select Advanced → Advanced Settings from the menu.<br />
<br />
682<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
c. Go to the Adapters and Bindings tab, as shown in Figure 19-8. d. Select a protocol and click either the up or down arrows to change the protocol binding order.<br />
<br />
Select a protocol and click either the up or down arrows to change the protocol binding order.<br />
<br />
Figure 19-8 Changing the protocol binding order in Windows 2000<br />
<br />
Use protocols that are efficient enough to handle the type of your network workload. As much as possible, keep your transport protocols to a minimum. Having too many protocols makes troubleshooting and maintenance difficult. Remove protocols not needed for network communications.To remove a protocol, follow removing network services instructions at “Software tuning options” on page 680 and select a network protocol in Windows instead. Move printers and users to other servers to lessen the load on your server. Network cabling hints include: – – – – – –<br />
<br />
Keeping 10Base-T devices to less than 35 per segment Using good-quality Category 5 cable in UTP installations Keeping Category 5 cable under the 100 meters total length limit Keeping 10Base2 segments to under 185 meters Keeping the token-ring devices to less than 75 per ring Checking for cabling faults, which causes frequent network slowdowns<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
683<br />
<br />
19.5.4 Monitoring network protocols Other than the performance objects and counters, it is also important to monitor how network protocols affect the network. Network protocols installed on your Windows 2000 affect the number of broadcasts and retransmissions. By monitoring the right counters for the protocols you selected, you can have a better understanding of the use of the network bandwidth.<br />
<br />
NetBEUI and NWLink NetBEUI uses broadcast to resolve NetBIOS names to the Media Access Control (MAC) physical address. It is a small, fast and low overhead protocol, but lacks routability and configuration options. It is best limited to a small LAN. Alternatively, NWLink is an IPX/SPX protocol stack designed for interpretability with Novell NetWare. NWLink is used in conjunction with a NetWare redirector such as Client Services for NetWare (CSNW). Both NetBEUI and NWLink have similar counters. Use the counters that are listed in Table 19-7 for monitoring. Table 19-7 NetBEUI counters<br />
<br />
Counter<br />
<br />
Description<br />
<br />
NetBEUI: Bytes Total/sec<br />
<br />
The total number of bytes sent in frames (data packets) and datagrams (such as broadcasts and acknowledgments). This value is normally high, indicating network utilization.<br />
<br />
NetBEUI: Datagrams/sec<br />
<br />
The number of non-guaranteed datagrams (broadcasts and acknowledgments) sent and received in the network. This value is normally high. Continually monitor the process to determine if it is causing excessive datagrams.<br />
<br />
NetBEUI: Frames/sec<br />
<br />
The number of data packets that have been sent and received in the network. If this value is abnormally high, indicating excessive broadcast, reduce your network broadcasts by employing name resolution strategies, such as DNS or WINS.<br />
<br />
684<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
TCP/IP TCP/IP in Windows Server 2003 includes a suite of tools that are common to most UNIX systems as well as TCP/IP supporting systems. TCP/IP supports open connectivity across hardware platforms and operating systems. It also supports routing for intranet and Internet applications. TCP/IP counters are added to a system when the TCP/IP protocol and the SNMP Service have been installed. The SNMP Service includes the objects and counters shown in Table 19-8 for TCP/IP related protocols. Table 19-8 TCP/IP counters<br />
<br />
Object<br />
<br />
Description<br />
<br />
TCP: Segments/sec<br />
<br />
The number of TCP segments (frames) that are sent and received over the network. This value is usually high, indicating high throughput.<br />
<br />
TCP: Segments Retransmitted/sec<br />
<br />
The number of frames (segments) that are retransmitted in the network. This value should be low. If sustained high values are observed, upgrade your physical hardware or segment your network.<br />
<br />
UDP: Datagrams/sec<br />
<br />
The number of UDP datagrams (such as broadcasts) that are sent and received.This value should be low. If sustained high values are observed, reduce your network broadcast.<br />
<br />
Network Interface: Output Queue Length<br />
<br />
The length of the output packet queue (in packets). Generally, a queue longer than two indicates congestion, and analysis of the network structure to determine the cause is necessary. This value should be low. If sustained high values are observed, upgrade the LAN adapter, add an additional LAN adapter, or verify the physical network components for failures.<br />
<br />
Chapter 19. Analyzing bottlenecks for servers running Windows<br />
<br />
685<br />
<br />
686<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
20<br />
<br />
Chapter 20.<br />
<br />
Analyzing bottlenecks for servers that are running Linux This chapter is useful if you are facing a reactive situation where a performance problem is already effecting a server. It presents a series of steps that lead to a concrete solution that you can implement to restore the server to an acceptable performance level. To manage your systems in a proactive manner, we suggest that you use the Capacity Manager. Capacity Manager for Linux allows you to monitor relevant server subsystems over a extended period of time. The topics that we cover in this chapter are: <br />
<br />
20.1, “Identifying bottlenecks” on page 688 20.2, “CPU bottlenecks” on page 692 20.3, “Memory subsystem bottlenecks” on page 694 20.4, “Disk bottlenecks” on page 698 20.5, “Network bottlenecks” on page 703<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
687<br />
<br />
20.1 Identifying bottlenecks We used the following steps as our quick tuning strategy: 1. 2. 3. 4. 5. 6.<br />
<br />
Know your system. Back up the system. Monitor and analyze the system performance. Narrow down the bottleneck and find its cause. Fix the cause of the bottleneck by trying a single change at a time. Go back to step 3 until you are satisfied with the performance of the system.<br />
<br />
Tip: You should document each step, especially the changes that you make and their effect on performance. Never change more than one factor at a time during your testing.<br />
<br />
20.1.1 Gathering information Most likely, the only first-hand information you have access to includes statements such as “There is a problem with the server.” In this situation, it is crucial that you use probing questions to clarify and to document the problem. Here is a list of questions that you should ask to obtain a better picture of the system: Could you give me a complete description of the server in question? – – – – –<br />
<br />
Model Age Configuration Peripheral equipment Operating system version and update level<br />
<br />
Can you describe the problem exactly? – What are the symptoms? – Describe any error messages. Some people can have problems answering this question. Any extra information might allow you to find the problem. For example, you might hear “It is really slow when I copy large files to the server.” Slow performance might indicate a network problem or a disk subsystem problem. Who is experiencing the problem? Is one person, one particular group of people, or the entire organization experiencing the problem? This type of question helps you to determine whether the problem exists in one particular part of the network, whether it is application-dependent, and so forth. If only one user is experiencing the<br />
<br />
688<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
problem, then the problem might be with the user’s PC (or the issue might be one of perception). The perception that clients have of the server is usually a key factor. From this point of view, performance problems might not be related directly to the server. The network path between the server and the clients can easily be the cause of the problem. This path includes network devices as well as services that are provided by other servers, such as domain controllers. Can the problem be reproduced? All reproducible problems can be solved. If you have sufficient knowledge of the system, you should be able to narrow the problem to its root and decide which actions to take. The fact that the problem can be reproduced allows you to see and to understand it better. Document the sequence of actions that are necessary to reproduce the problem at any time: – What are the steps to reproduce the problem? Knowing the steps might let you reproduce the same problem on a different machine under the same conditions. If this works, it gives you the opportunity to use a machine in a test environment and removes the chance of crashing the production server. – Is it an intermittent problem? If the problem is intermittent, the first thing to do is to gather information and find a path to move the problem in the reproducible category. The goal here is to have a scenario to make the problem happen on command. – Does it occur at certain times of the day or certain days of the week? This information might help you to determine what is causing the problem. The problem might occur when everyone arrives for work or returns from lunch. Look for ways to change the timing (that is, make it happen less or more often). It there are ways to do so, the problem becomes a reproducible one. – Is it unusual? If the problem falls into the non-reproducible category, you can conclude that it is the result of extraordinary conditions and classify it as fixed. In real life, there is a high probability that it will happen again. A good procedure to troubleshoot a hard-to-reproduce problem is to perform general maintenance on the server: reboot or bring the machine up-to-date on drivers and patches.<br />
<br />
Chapter 20. Analyzing bottlenecks for servers that are running Linux<br />
<br />
689<br />
<br />
When did the problem start? Was it gradual or did it occur very quickly? If the performance issue appeared gradually, then it is likely to be a sizing issue. If it appeared overnight, then the problem could be caused by a change made to the server or peripherals. Have any changes been made to the server (minor or major), or are there any changes in the way clients are using the server? Did the customer alter something on the server or peripherals to cause the problem? Is there a log of all network changes available? Demands could change based on business changes, which could affect demands on a servers and network systems, for example: Are there any other servers or hardware components involved? Are there any logs available? What is the priority of the problem? When does it need to be fixed? – Does it need to be fixed in the next few minutes, or in days? You might have some time to fix it, or it might already be time to operate in panic mode. – How massive is the problem? – What is the related cost of that problem?<br />
<br />
20.1.2 Analyzing the server’s performance Important: Before taking any troubleshooting actions, back up all data and the configuration information to prevent a partial or complete lost. At this point, you should begin monitoring the server. The simplest way to monitor the server is to run monitoring tools from the server that you are analyzing. An alternative is to gather measurements that are generated with IBM Director’s Capacity Manager, which continually records data in the background. For more information about how to use Capacity Manager, refer to Chapter 17, “Capacity Manager” on page 591.<br />
<br />
690<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
You should create a performance log of the server during its peak time of operation (for example, 9:00 a.m. to 5:00 p.m.). The peak performance time for your server will depend upon what services are provided and who is using these services. When creating the log, if available, include the following objects: <br />
<br />
Processor System Server work queues Memory Page file Physical disk Redirector Network interface<br />
<br />
Before you begin, remember that a methodical approach to performance tuning is important. Our recommended process, which you can use for your System x server performance tuning process, is as follows: 1. Understand the factors that effect server performance as explained in the first chapters of this redbook. Especially try to understand the logical connection of the various subsystems and the distinction between a hardware and a software bottleneck. 2. Measure the current performance to create a performance baseline to compare with your future measurements and to identify system bottlenecks. 3. Use available monitoring tools to identify a performance bottleneck. By following the instructions in this chapter, you should be able to narrow down the bottleneck to the subsystem level. 4. Improve the component that is causing the bottleneck by performing some actions to improve server performance in response to demands. Note: It is important to understand that the greatest gains are obtained by upgrading a component that has a bottleneck when the other components in the server have ample power left to sustain an elevated level of performance. 5. Measure the new performance so that you can compare the performance before and after the tuning steps. When attempting to fix a performance problem, remember the following: Take measurements before you upgrade or modify anything so that you can tell whether the change had any effect (that is, take baseline measurements). Examine the options that involve reconfiguring existing hardware, not just those that involve adding new hardware.<br />
<br />
Chapter 20. Analyzing bottlenecks for servers that are running Linux<br />
<br />
691<br />
<br />
20.2 CPU bottlenecks For servers whose primary role is that of an application or database server, the CPU is a critical resource and can often be a source of performance bottlenecks. It is important to note that high CPU utilization does not always mean that a CPU is busy doing work. It might, in fact, be waiting on another subsystem. When performing proper analysis, it is very important that you look at the system as a whole and look at all subsystems because there can be a cascade effect within the subsystems. Note: There is, however, a common misconception that the CPU is the most important part of the server. Unfortunately, this is often not the case and, as such, servers are often over configured with CPU and under configured with disks, memory, and network subsystems. Only specific applications that are truly CPU-intensive can take advantage of today’s high-end processors.<br />
<br />
20.2.1 Finding bottlenecks with the CPU Determining bottlenecks with the CPU can be accomplished in several ways. As we have discussed in Chapter 15, “Linux tools” on page 537, Linux has a variety of tools to help determine CPU bottlenecks. The question is which tools should you use? One such tool is uptime. By analyzing the output from uptime, you can get a rough idea of what has happened in the system for the last 15 minutes (Example 20-1). For a more detailed explanation of this tool, see 15.1, “The uptime command” on page 539. Example 20-1 The uptime tool output from a CPU strapped system<br />
<br />
18:03:16 up 1 day, 2:46, 6 users, load average: 182.53, 92.02, 37.95 Using KDE System Guard and the CPU sensors lets you view the current CPU workload. Tip: Be careful not to add to CPU problems by running too many tools at one time. You might find that using a lot of different monitoring tools at one time can contribute to the high CPU load. Also, keep in mind that X-based monitoring tools bring some overhead with them due to the GUI. Never attempt to measure a imminent memory bottleneck with the aid of GUI-based tools, because they increase memory demand even further.<br />
<br />
692<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Using top, you can see CPU utilization and also what processes are the biggest contributor to the problem, as shown in Example 15-3 on page 541. If you have set up sar, you are collecting a lot of information, some of which is CPU utilization over a period of time. Analyzing this information can be difficult, so use isag, which can take sar output and plot a graph. Otherwise, you might want to parse the information through a script and use a spreadsheet to plot it to see any trends in CPU utilization. You can also use sar from the command line by issuing sar -u or sar -U processor-number. To gain a broader perspective of the system and current utilization of more than just the CPU subsystem, a good tool is vmstat, which is described in greater detail in 15.5, “The vmstat command” on page 545.<br />
<br />
20.2.2 Multi-processing machines Issues with multi-processing machines can be difficult to detect. In an SMP environment, the concept of CPU affinity implies that you bind a process to a CPU. CPU affinity is useful with CPU cache optimization, which is achieved by keeping the same process on one CPU rather than moving the process between processors. When a process moves between CPUs, the cache of the new CPU must be flushed. So, a process that moves between processors causes many cache flushes to occur. Thus, an individual process takes longer to finish. This scenario is very difficult to detect because it appears that the CPU load is very balanced and that it is not necessarily peaking on any CPU. Affinity is also useful in NUMA-based systems, such as servers based on the AMD Opteron and the System x 3850 and System x 3950, where it is important to keep memory, cache, and CPU access local to one another.<br />
<br />
20.2.3 Performance tuning options for the CPU When attempting to tune the CPU, you should first ensure that the system performance problem is caused by the CPU and not one of the other subsystems. If it is the processor that is the server bottleneck, then you can take a number of steps to improve performance, including: Ensure that no unnecessary programs are running in the background by using ps -ef. If you find unnecessary programs that are running, stop these programs and use cron to schedule them to run at off-peak hours. Identify non-critical, CPU-intensive processes by using top and modify their priority using renice. In an SMP-based machine, try using taskset to bind processes to CPUs to make sure that processes are not hopping between processors and causing cache flushes.<br />
<br />
Chapter 20. Analyzing bottlenecks for servers that are running Linux<br />
<br />
693<br />
<br />
Based on the application that is running, decide whether it is better to scale up (bigger CPUs) than scale out (more CPUs). This decision is a function of whether your application was designed to take advantage of more processors effectively. For example, a single-threaded application scales better with a faster CPU and not with more CPUs. Ensure that sure you are using the latest drivers and firmware, which can affect the load on the CPU.<br />
<br />
20.3 Memory subsystem bottlenecks On a Linux system, many programs run at the same time. These programs support multiple users, and some processes are more used than others. Some of these programs use a portion of memory while the rest are “sleeping.” When an application accesses cache, the performance increases because an in-memory access retrieves data, thereby eliminating the need to access slower disks. The operating system uses an algorithm to control which programs use physical memory and which programs are paged out. This paging of memory is transparent to user programs. Page space is a file created by the operating system on a disk partition to store user programs that are not used currently. Typically, page sizes are 4 KB or 8 KB. In Linux, the page size is defined in the kernel header file include/asm-<architecture>/param.h, using the variable EXEC_PAGESIZE. The process that is used to page out a process to disk is called pageout.<br />
<br />
20.3.1 Finding bottlenecks in the memory subsystem To find bottlenecks in the memory subsystem, you should start your analysis by listing the applications that are running on the server. Determine how much physical memory and swap each of the applications needs to run. Figure 20-1 on page 695 shows KDE System Guard monitoring memory usage.<br />
<br />
694<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 20-1 KDE System Guard memory monitoring<br />
<br />
The indicators in Table 20-1 can also help you define a problem with memory. Table 20-1 Indicator for memory analysis<br />
<br />
Memory indicator<br />
<br />
Analysis<br />
<br />
Memory available<br />
<br />
Indicates how much physical memory is available for use. If, after you start your application, this value decreases significantly, you might have a memory leak. Check the application that is causing it and make the necessary adjustments. Use free -l -t -o for additional information.<br />
<br />
Page faults<br />
<br />
There are two types of page faults: soft page faults, when the page is found in memory, and hard page faults, when the page is not found in memory and must be fetched from disk. Accessing the disk slows down your application considerably. The sar -B command can provide useful information for analyzing page faults, specifically columns pgpgin/s and pgpgout/s.<br />
<br />
File system cache<br />
<br />
This is the common memory space used by the file system cache. Use the free -l -t -o command, for example.<br />
<br />
Private memory for process<br />
<br />
Represents the memory that is used by each process running on server. You can see how much memory is allocated to specific process using the pmap command.<br />
<br />
Chapter 20. Analyzing bottlenecks for servers that are running Linux<br />
<br />
695<br />
<br />
Paging and swapping indicators In Linux, as with all UNIX-based operating systems, there are differences between paging and swapping. Paging moves individual pages to swap space on the disk. Swapping is a bigger operation that moves the entire address space of a process to swap space in one operation. Swapping can have one of two causes: A process enters sleep mode. Sleep node normally happens because the process depends on interactive action, because editors, shells, and data entry applications spend most of their time waiting for user input. During this time, they are inactive. A process behaves poorly. Paging can be a serious performance problem when the amount of free memory pages falls below the minimum amount specified, because the paging mechanism is not able to handle the requests for physical memory pages and the swap mechanism is called to free more pages. This type of paging increases I/O to disk significantly and degrades a server’s performance quickly. If your server is always paging to disk (a high page-out rate), consider adding more memory. However, for systems with a low page-out rate, adding memory might not have any effect.<br />
<br />
Using NMON NMON displays available physical memory, low space memory, high memory, and swap space in total, free, and free percentage. With NMON, you can view very quickly whether your system is swapping by looking at the free percentage value of the swap memory column. You can also see whether your server is running out of memory or whether the applications or the operating system is consuming too much memory. To display the memory statistics in NMOM, press the m key. Figure 20-2 on page 697 illustrates the memory monitoring with NMON.<br />
<br />
696<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 20-2 NMON monitoring memory<br />
<br />
20.3.2 Performance tuning options for the memory subsystem It you think that there is a memory bottleneck, consider these actions: Tune the swap space using bigpages, hugetlb, or shared memory. Increase or decrease the size of pages. Improve the handling of active and inactive memory. Adjust the page-out rate. Limit the resources that are used for each user on the server. Stop the services that you do not need as discussed in 12.1, “Disabling daemons” on page 372. Add memory.<br />
<br />
Chapter 20. Analyzing bottlenecks for servers that are running Linux<br />
<br />
697<br />
<br />
20.4 Disk bottlenecks The disk subsystem is often the most important aspect of server performance and is usually the most common bottleneck. However, problems can be hidden by other factors, such as lack of memory. Applications are considered to be I/O bound when CPU cycles are wasted simply waiting for I/O tasks to finish. The most common disk bottleneck is having too few disks. Most disk configurations are based on capacity requirements, not performance. The least expensive solution is to purchase the smallest number of the largest-capacity disks possible. However, this places more user data on each disk, causing greater I/O rates to the physical disk and allowing disk bottlenecks to occur. The second most common problem is having too many logical disks on the same array, which increases seek time and greatly lowers performance. We discuss the disk subsystem in 12.8, “Tuning the file system” on page 395.<br />
<br />
20.4.1 Finding bottlenecks in the disk subsystem A server that exhibits the following symptoms might be suffering from a disk bottleneck (or a hidden memory problem): Slow disks result in memory buffers that fill with write data or that wait for read data, which delays all requests because free memory buffers are unavailable for write requests. Alternatively, the response waits for read data in the disk queue or there is insufficient memory because there is not enough memory buffers for network requests, which can cause synchronous disk I/O. Disk or controller use is typically very high. Most LAN transfers happen only after disk I/O has completed, which causes very long response times and low network utilization. Because disk I/O can take a relatively long time and disk queues can become full, the CPUs are idle or have low utilization because they wait a long time before processing the next request. The disk subsystem is perhaps the most challenging subsystem to configure properly. In addition to looking at raw disk interface speed and disk capacity, it is key to also understand the workload. Is disk access random or sequential? Is there large I/O or small I/O? Answering these questions can provide the necessary information to make sure that the disk subsystem is tuned adequately. Disk manufacturers tend to showcase the upper limits of their drive technology’s throughput. However, taking the time to understand the throughput of your workload can help you to set true expectations for your underlying disk subsystem.<br />
<br />
698<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Table 20-2 Exercise showing true throughput for 8K I/Os for different drive speeds<br />
<br />
Disk Speed<br />
<br />
Latency<br />
<br />
Seek Time<br />
<br />
Total Random Access Timea<br />
<br />
I/Os per second per diskb<br />
<br />
Throughput given 8 KB I/O<br />
<br />
15 000 RPM<br />
<br />
2.0 ms<br />
<br />
3.8 ms<br />
<br />
6.8 ms<br />
<br />
147<br />
<br />
1.15 MBps<br />
<br />
10 000 RPM<br />
<br />
3.0 ms<br />
<br />
4.9 ms<br />
<br />
8.9 ms<br />
<br />
112<br />
<br />
900 KBps<br />
<br />
7 200 RPM<br />
<br />
4.2 ms<br />
<br />
9 ms<br />
<br />
13.2 ms<br />
<br />
75<br />
<br />
600 KBps<br />
<br />
a. b.<br />
<br />
Assuming that the handling of the command + data transfer < 1 ms, total random access time = latency + seek time + 1 ms. Calculated as 1/total random access time.<br />
<br />
Random read/write workloads usually require several disks to scale. The bus bandwidths of SCSI or Fibre Channel are of lesser concern. Larger databases with random access workload benefit from having more disks. Larger SMP servers scale better with more disks. Given the I/O profile of 70% reads and 30% writes of the average commercial workload, a RAID-10 implementation performs 50% to 60% better than a RAID-5. Sequential workloads tend to stress the bus bandwidth of disk subsystems. Pay special attention to the number of SCSI buses and Fibre Channel controllers providing a greater connection bandwidth where maximum throughput is desired. Given the same number of drives in an array, RAID-10, RAID-0, and RAID-5, all have similar streaming read and write throughput. There are two ways to approach disk bottleneck analysis: Real-time monitoring must be done while the problem is occurring. Real-time monitoring might not be practical in cases where system workload is dynamic and the problem is not repeatable. However, if the problem is repeatable, this method is very flexible because of the ability to add objects and counters as the problem becomes well understood. Tracing is the collecting of performance data over time to diagnose a problem. This method is a good way to perform remote performance analysis. Some of the drawbacks of this method include the potential for having to analyze large files when performance problems are not repeatable and the potential for not having all the key objects or parameters in the trace and having to wait for the next time the problem occurs for the additional data.<br />
<br />
The vmstat command You can use the vmstat tool to track disk usage on a Linux system. The columns of interest in vmstat with respect to I/O are the bi and bo fields. These fields monitor the movement of blocks in and out of the disk subsystem. Having a baseline is key to being able to identify any changes over time.<br />
<br />
Chapter 20. Analyzing bottlenecks for servers that are running Linux<br />
<br />
699<br />
<br />
Example 20-2 shows an example of vmstat output. Example 20-2 vmstat output [root@x232 root]# vmstat 2 r b swpd free buff 2 1 0 9004 47196 0 2 0 9672 47224 0 2 0 9276 47224 0 2 0 9160 47224 0 2 0 9272 47224 0 2 0 9180 47228 1 0 0 9200 47228 1 0 0 9756 47228 0 2 0 9448 47228 0 2 0 9740 47228<br />
<br />
cache si so bi bo in cs us sy id 1141672 0 0 0 950 149 74 87 13 1140924 0 0 12 42392 189 65 88 10 1141308 0 0 448 0 144 28 0 0 1141424 0 0 448 1764 149 66 0 1 1141280 0 0 448 60 155 46 0 1 1141360 0 0 6208 10730 425 413 0 3 1141340 0 0 11200 6 631 737 0 6 1140784 0 0 12224 3632 684 763 0 11 1141092 0 0 5824 25328 403 373 0 3 1140832 0 0 640 0 159 31 0 0<br />
<br />
wa 0 0 0 1 0 100 0 99 0 99 0 97 0 94 0 89 0 97 0 100<br />
<br />
The iostat command You can also encounter performance problems when too many files are opened and read and written to, and then closed repeatedly. This problem can become apparent as seek times (the time it takes to move to the exact track where the data is stored) start to increase. Using the iostat tool, you can monitor the I/O device loading in real time. Different options for this tool allow you to drill down even further to gather the necessary data. Example 20-3 shows a potential I/O bottleneck on the device /dev/sdb1. This output shows average wait times (await) of around 2.7 seconds and service times (svctm) of 270 ms. Example 20-3 Sample of an I/O bottleneck as shown with iostat 2 -x /dev/sdb1 [root@x232 root]# iostat 2 -x /dev/sdb1 avg-cpu:<br />
<br />
%user 11.50<br />
<br />
%nice 0.00<br />
<br />
%sys 2.00<br />
<br />
Device: rrqm/s wrqm/s await svctm %util /dev/sdb1 441.00 3030.00 2717.33 266.67 100.00<br />
<br />
r/s<br />
<br />
avg-cpu:<br />
<br />
%sys 1.00<br />
<br />
%user 10.50<br />
<br />
%nice 0.00<br />
<br />
Device: rrqm/s wrqm/s await svctm %util /dev/sdb1 441.00 3030.00 2739.19 270.27 100.00<br />
<br />
700<br />
<br />
%idle 86.50 w/s<br />
<br />
rsec/s<br />
<br />
wsec/s<br />
<br />
7.00 30.50 3584.00 24480.00<br />
<br />
r/s<br />
<br />
rkB/s<br />
<br />
wkB/s avgrq-sz avgqu-sz<br />
<br />
1792.00 12240.00<br />
<br />
748.37<br />
<br />
101.70<br />
<br />
%idle 88.50 w/s<br />
<br />
rsec/s<br />
<br />
wsec/s<br />
<br />
7.00 30.00 3584.00 24480.00<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
rkB/s<br />
<br />
wkB/s avgrq-sz avgqu-sz<br />
<br />
1792.00 12240.00<br />
<br />
758.49<br />
<br />
101.65<br />
<br />
avg-cpu:<br />
<br />
%user 10.95<br />
<br />
%nice 0.00<br />
<br />
Device: rrqm/s wrqm/s await svctm %util /dev/sdb1 438.81 3165.67 2728.00 268.00 100.00<br />
<br />
%sys 1.00 r/s<br />
<br />
%idle 88.06 w/s<br />
<br />
rsec/s<br />
<br />
wsec/s<br />
<br />
6.97 30.35 3566.17 25576.12<br />
<br />
rkB/s<br />
<br />
wkB/s avgrq-sz avgqu-sz<br />
<br />
1783.08 12788.06<br />
<br />
781.01<br />
<br />
101.69<br />
<br />
The iostat -x command (for extended statistics) provides low-level detail of the disk subsystem. The output for this command gives you the following information: %util: percentage of CPU consumed by I/O requests svctm: average time required to complete a request, in milliseconds await: average amount of time an I/O waited to be served, in milliseconds avgqu-sz: average queue length avgrq-sz: average size of request rrqm/s: the number of read requests merged per second that were issued to the device wrqm/s: the number of write requests merged per second that were issued to the device For a more detailed explanation of the fields, see the man page for iostat. Changes made to the elevator algorithm as described in “Tuning the elevator algorithm (kernel 2.4 only)” on page 399, will be seen in the avgrq-sz (average size of request) and avgqu-sz (average queue length). Because the latencies are lowered by manipulating the elevator settings, the avgrq-sz will go down. You can also monitor the rrqm/s and wrqm/s to see the effect on the number of merged reads and writes that the disk can manage.<br />
<br />
Using NMON When using NMON, you can view the disk activity. It gives you a quick overview of the disks and partitions throughput in KBps. To display the disks activity with NMOM, press the d key. NMON also displays what kind of I/O is performed (read or write), as well as the disks utilization. The refreshing period is user-defined, and you can monitor the activity of the disks in timed intervals (every two seconds, for example). Figure 20-3 on page 702 illustrates the NMON display with, among other information, the disks activity.<br />
<br />
Chapter 20. Analyzing bottlenecks for servers that are running Linux<br />
<br />
701<br />
<br />
Figure 20-3 NMON monitoring disks<br />
<br />
20.4.2 Performance tuning options for the disk subsystem After verifying that the disk subsystem is a bottleneck, a number of solutions are possible, including: If the workload is of a sequential nature and it is stressing the controller bandwidth, the solution is to add a faster disk controller. However, if the workload is more random in nature, then the bottleneck is likely to involve the disk drives, and adding more drives will improve performance. Add more disk drives in a RAID environment to spread the data across multiple physical disks and improve performance for both reads and writes. This addition increases the number of I/Os per second. Also, use hardware RAID instead of the software implementation that is provided by Linux. If hardware RAID is used, the RAID level is hidden from the operating system.<br />
<br />
702<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Offload processing to another system in the network (either users, applications, or services). Add more RAM. Adding memory increases system memory disk cache, which in effect improves disk response times.<br />
<br />
20.5 Network bottlenecks A performance problem in the network subsystem can be the cause of many problems, such as a kernel panic. To analyze these anomalies to detect network bottlenecks, each Linux distribution includes traffic analyzers.<br />
<br />
20.5.1 Finding network bottlenecks We recommend KDE System Guard because of its graphical interface and ease of use. The tool is also available on the distribution CDs. For more information about this tool, see 15.7, “KDE System Guard” on page 547.<br />
<br />
Figure 20-4 KDE System Guard network monitoring<br />
<br />
Chapter 20. Analyzing bottlenecks for servers that are running Linux<br />
<br />
703<br />
<br />
For SUSE Linux, you can also use the traffic-vis package, an excellent network monitoring tool. You capture the network data then analyze the results using a Web browser. For details, see 15.9, “Traffic-vis” on page 554. It is important to remember that there are many possible reasons for these performance problems and that sometimes, problems occur simultaneously, making it even more difficult to pinpoint the origin. The indicators in Table 20-3 can help you determine the problem with your network: Table 20-3 Indicators for network analysis<br />
<br />
Network indicator<br />
<br />
Analysis<br />
<br />
Packets received Packets sent<br />
<br />
Shows the number of packets that are coming in and going out of the specified network interface. Check both internal and external interfaces.<br />
<br />
Collision packets<br />
<br />
Collisions occur when there are many systems on the same domain. The use of a hub might be the cause of many collisions.<br />
<br />
Dropped packets<br />
<br />
Packets can be dropped for a variety of reasons, but the result might impact performance. For example, if the server network interface is configured to run at 100 Mbps full duplex, but the network switch is configured to run at 10 Mbps, the router might have an ACL filter that drops these packets, for example: iptables -t filter -A FORWARD -p all -i eth2 -o eth1 -s 172.18.0.0/24 -j DROP<br />
<br />
Errors<br />
<br />
Errors occur if the communications lines (for example, the phone line) are of poor quality. In these situations, corrupted packets must be present, thereby decreasing network throughput.<br />
<br />
Faulty adapters<br />
<br />
Network slowdowns often result from faulty network adapters. When this kind of hardware fails, it can begin to broadcast junk packets in the network.<br />
<br />
20.5.2 Performance tuning options for the network subsystem To solve problems related to network bottlenecks: Ensure that the network card configuration matches router and switch configurations (for example, frame size). Modify how your subnets are organized. Use faster network cards.<br />
<br />
704<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Tune the appropriate IPV4 TCP kernel parameters. See Chapter 12, “Linux” on page 371 for more information. Some security-related parameters can also improve performance, as described in that chapter. If possible, change network cards and check performance again. Add network cards and bind them together to form an adapter team, if possible.<br />
<br />
Chapter 20. Analyzing bottlenecks for servers that are running Linux<br />
<br />
705<br />
<br />
706<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21<br />
<br />
Chapter 21.<br />
<br />
Case studies In this chapter, we present four case studies for servers with performance problems. In these studies, we show how the subsystems behave for different load conditions and how to detect bottlenecks using monitoring tools. We also analyze the data that is generated by these tools, arrive at a conclusion based on this analysis, and recommend a course of action to improve performance. We begin our discussion with a general overview of the two modes for system monitoring. We then discuss the following case studies: <br />
<br />
21.1, “Analyzing systems” on page 708 21.2, “SQL Server database server” on page 709 21.3, “File servers hang for several seconds” on page 722 21.4, “Database server” on page 729 21.5, “ERP application server” on page 737<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
707<br />
<br />
21.1 Analyzing systems To analyze a system, it is best to start with an overview and then move down in details to the bottleneck components. Essentially, you can divide bottleneck components into hardware, software, and external (users and network). There are two modes for monitoring a system: trace and real-time.<br />
<br />
Trace mode Trace mode allows you to collect data for a given period of time, so that you can monitor the system during a specific activity period. You should collect all possible counters so that there is no need to re-run the workload if a counter is forgotten. You should then collect all counters for the following objects: <br />
<br />
Processor Memory Physical disk Network System Processor queue length<br />
<br />
Do not trace if it is not necessary, because doing so can consume space. For advanced software bottlenecks, you can add the following objects: Process Thread<br />
<br />
Real-time monitoring Real-time monitoring is useful while a problem is actually occurring. This mode requires a steady state problem that is occurring on a server with no dynamic workload. Otherwise, it can be difficult to isolate the bottlenecks. This mode allows you to add and to remove any counter at any time. So, you can examine each counter as you begin to understand the problem. However, it is important to keep in mind that Perfmon displays averages for most counters. We made all of the case studies in this chapter using Perfmon. Perfmon is the default and most commonly used performance monitoring tool for Windows systems. However, you can also use Perfmon to display Linux performance logs when used with xPL. For information about Perfmon, see Chapter 14, “Windows tools” on page 471, and for information about for xPL, see Chapter 15, “Linux tools” on page 537.<br />
<br />
708<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.2 SQL Server database server This case study does not focus on a particular behavior from the system. It is more an analysis of how the server is actually working and what can be done to improve overall performance. The server in this case is a 4-way Xeon. Figure 21-1 illustrates the maximum values overview in Perfmon. These values provide a good view of the data and let us detect multiple bottlenecks. They also show an example of trace mode system monitoring.<br />
<br />
n o<br />
<br />
p q Figure 21-1 Maximum values overview<br />
<br />
In Figure 21-1, the Page Reads and Writes per second counter for memory n. It indicates a moderate value that might imply a memory bottleneck. As well, the values indicate that 7.6 MBps have been received from the network interface, o. Given that this is a 100 Mbps Ethernet adapter, this value is high. Moreover, there are bottlenecks on the disks shown here. More that four seconds as an average disk seconds per write (latency) p indicate a bottleneck. Finally, the percentage processor time on each CPU is 100% (q) because every CPU is saturated. Let us analyze each of these components more thoroughly.<br />
<br />
21.2.1 Memory analysis Memory is analyzed using the paging counters and the cache hit ratio. This analysis indicates if there is enough installed memory and if some data is written to or read on disks instead of on memory. A trace mode monitoring of the<br />
<br />
Chapter 21. Case studies<br />
<br />
709<br />
<br />
memory subsystem provides some information about how memory is accessed and used by SQL Server. Figure 21-2 shows that only 0.3 pages are read per second as an average, while the peak is reaching 142 page reads (white lines). This information tells us the paging activity is not a primary bottleneck but is effecting performance negatively. Additional checks of SQL Server Buffer Cache hit rate might be necessary in order to determine whether you can take away memory or add additional memory from SQL Server to reduce paging.<br />
<br />
Figure 21-2 Memory analysis<br />
<br />
710<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 21-3 shows cache faults per second. Cache faults indicate that server cache pages were requested but not available in cache (probably because of low cache memory capacity). Combined with page accesses (read and write) per second, this analysis indicates that these pages are being accessed on the paging disk. The server is then short on operating system and application memory.<br />
<br />
Figure 21-3 Cache faults per second<br />
<br />
From this data, it appears that the server is acting as a file server or running some application that is using memory outside of SQL Server. This activity is producing cache page misses and pressuring the reserved memory for the cache. For optimal performance, you should add another 1 GB or 2 GB of memory for operating system use.<br />
<br />
21.2.2 Processor analysis The average processor utilization in more than 40%, with frequent peaks at 100%. This usage is divided in privileged time with an average of 12%, maximum at 67% and a user time, which is the application usage. This application time represents the majority of the CPU’s utilization.<br />
<br />
Chapter 21. Case studies<br />
<br />
711<br />
<br />
Figure 21-4 shows that regular peaks at 100% of the processor utilization are occurring. However, interrupt and privileged time percentages indicate optimal operating system and drivers efficiency. Therefore, for optimal performance, it might be necessary to offload some workload from the server, to upgrade the server with faster processors, or even to replace the server with a new server with faster processors.<br />
<br />
Figure 21-4 Processor time<br />
<br />
712<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.2.3 Network analysis Figure 21-5 shows an average counters analysis and Figure 21-6 shows the corresponding maximums from the trace logs on the network subsystem. From this analysis, the average throughput for the only network adapter is around 2 Mbps with a maximum reaching 84 Mbps. For a 100 Mbps network connection, the maximum value is very close to the limit. However, the average throughput (read and write) is very low, so you need to investigate the network subsystem further. Further analysis shows that the network is experiencing heavy receive traffic and almost no send traffic. The receive traffic itself is not linear and peaks occur during a third of the run period, which reduces the average value dramatically.<br />
<br />
Figure 21-5 Average network utilization<br />
<br />
Figure 21-6 Maximum network utilization<br />
<br />
Chapter 21. Case studies<br />
<br />
713<br />
<br />
The peak receive traffic during a period of time might show a network limitation. Indeed, the maximum values are almost reached, and a network subsystem improvement would increase the upper limit. Moreover, if the other components are improved (memory and processor, for example), the throughput demand might increase, and the network might then act as a bottleneck. Thus, you could implement the following solutions to improve the network performance: Add a new adapter and set up adapter teaming. Upgrade to a 1 Gbps connection. (Note that you must upgrade the remainder of the network components as well, such as switches and other servers adapters.) Upgrade the current version of the operating system from Windows 2000 to Windows Server 2003, because Windows Server 2003 has an improved network stack that is capable of sustaining about 90% to 95% utilization (compared to 75% to 80% for Windows 2000).<br />
<br />
714<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.2.4 Disk analysis on the C: drive The C: drive is used to host the operating system and the application program. As well, all Perfmon operations (such as logs) are done on the C: drive. The optimal performance for a given disks subsystem is obtained when latency is not more than 25 ms to 30 ms. Figure 21-7 shows the C: drive read latency. The average read latency is around 7 ms, which is good, compared to the 25 ms to 30 ms. Some peak values sometimes reach 60 ms, which is moderate.<br />
<br />
Figure 21-7 C: drive read latency<br />
<br />
Chapter 21. Case studies<br />
<br />
715<br />
<br />
Figure 21-8 shows write latency on the same disk. The data shows an average latency around 6 ms (very good), but important peaks at more than 2.5 seconds. Some frequent write latency peaks indicate a significant bottleneck.<br />
<br />
Figure 21-8 C: drive write latency<br />
<br />
The write latency correlates with the page write activity, which can be significantly reduced by adding more memory to the system. Of course, Perfmon writes some log onto the disk and, therefore, can increase the write latency. However, this behavior is insignificant compared to the overall performance. Further analysis shows that the Average “Disk Bytes per Read” (which is the read size) is around 4 KB with peaks at 64 KB. As well, the drive write size gives frequent maximum values at 64 KB. The default stripe size for that disk is 8 KB. For performing a 64 KB read or write access, the system actually runs eight times an 8 KB block I/O, because of the 8 KB stripe size. This analysis shows that reformatting the C: drive to use a 64 KB stripe size will improve performance by reducing latency. Finally, disk throughput analysis shows an average read throughput around 4 KBps, with a maximum of only 1.5 MBps and an average write throughput of 90 KBps with a maximum of 5 MBps. These values are very low, for a RAID-1 array of two SCSI disks. The bandwidth SCSI disks can reach is higher than 250 MBps. Consequently, we can say there is absolutely no disk throughput bottleneck on the C: drive. Considering that the operating system is installed on that disk and that nothing but disk paging is occurring on that disk, these results are normal and expected.<br />
<br />
716<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.2.5 Disk analysis on the D: drive The D: drive is used for logs and as temporary data store (tempDB). Latency monitoring on that disk shows extremely high read and write values (up to 4.3 seconds). Additional analysis on the disk’s stripe size show an average read size of 66 KB with a maximum of 262 KB, and an average write size of 51 KB with a maximum of 521 KB (as shown in Figure 21-9).<br />
<br />
Figure 21-9 Read/write strip sizes on disk D:<br />
<br />
In addition, I/O analysis shows very heavy load on write accesses with an average of 125 operations per second and a maximum of 1148 operations per second. With a default stripe size set at 8 KB, there are too many physical operations occurring on the disk array. Given that the array is based on four disks, the maximum number of operations per second is very high. You can reduce the number of operations per second by doing the following: Increasing the number of disks (to dispatch the load) Increasing the stripe size to 64 KB or more Replacing the disks with faster disks (15K RPM instead of 10K RPM)<br />
<br />
Chapter 21. Case studies<br />
<br />
717<br />
<br />
The analysis shows that any accesses are occurring on the disk array and that these accesses take a very long time to complete. As a consequence, the throughput is quite low. Indeed, the D: drive has an average read throughput of 3.1 MBps (peaks at 32 MBps) and an average write throughput at 6.4 MBps (peaks at 57 MBps). Therefore, we cannot say there is a throughput bottleneck, even if the disk array is acting as one. It is more the latency and the stripe size that matter here.<br />
<br />
21.2.6 Disk analysis of the V: drive The V: disk is used for the database. When the latency on disk V: is monitored, we notice an average read latency around 20 ms, with a maximum of 647 ms. Whereas the average is acceptable, the peaks are very high. So, you should analyze the disk further. In addition, the average write latency is 29 ms with extremely high peaks at 4 seconds. Further analysis shows an average read size of 12.4 KB with some peaks at 94 KB and an average write size of 51 KB with frequent peaks at 538 KB. However, most of the write peaks are under 64 KB. Thus, the long write latency that is associated with large and frequent random writes produces a bottleneck on the drive. Again, the default stripe size is here 8 KB, which generates eight physical disk I/O for the frequent 64 KB writes, which partially explains the long latency. A deeper analysis of I/O performance (as shown in Figure 21-10 on page 719) shows a disk read operation per second that is around 35 as an average and some 935 operations per second as maximum peaks for read. The load is moderate for a six disk array. Additionally, the write operations are in average at 52 operations per second and reaches peaks at 1584 operations per second.<br />
<br />
718<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 21-10 Disk V: I/Os<br />
<br />
When looking at the combined read/write rate, we see an average of 87 I/Os per second (35 read operations + 52 write operations) and a peak of 1584 operations per second, which indicates only write load for peaks. Given that the array of six disks can deliver at most 200 to 250 I/O per second for each disk, the maximum operations per second that this array could perform is around 1200 to 1500 I/O per second. This limit is passed with the write peaks (see the bottom graph in Figure 21-10). The heavy I/O load on the V: disk array, coupled with very large writes, causes multiple physical I/O and long latency. The system is then bottlenecked on that drive. To improve performance, you should increase the stripe size up to 64 KB and at least double the number of drives. The throughput analysis shows light to moderate throughput on the V: drive with an average read about 602 KBps (peaks at 49 MBps). The write transfer rates have an average of 2.7 MBps and peaks at 38 MBps. The SCSI bus can support up to 320 MBps throughput and, therefore, there is no throughput bottleneck on the drive.<br />
<br />
Chapter 21. Case studies<br />
<br />
719<br />
<br />
21.2.7 SQL Server analysis Figure 21-11 illustrates that free pages are sometimes dropping to zero. The average free pages is 5246 pages (or 41.9 MB), which is very low. In addition, the Procedure cache is consuming all memory. As a consequence, starving the procedure cache can cause excess I/O to tempDB and might explain the high read rate to D: drive.<br />
<br />
Figure 21-11 SQL Server buffer trace logs<br />
<br />
Adding 2 GB of memory dedicated to SQL server would improve performance in this situation.<br />
<br />
720<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.2.8 Summary All subsystems of this server are considered bottlenecks mainly because a single component is having issues in performing the I/Os and, thus, the entire system is impacted. In summary, the trace logs of Perfmon indicate: The memory subsystem is the greatest bottleneck in the system, experiencing excessive paging and memory starvation while heavy paging is associated with slow disk performance for the C: drive. It appears that the server is acting as a file server or running some application that is using memory outside of SQL Server. This activity is producing cache page misses and pressuring the reserved memory for the cache. For optimal performance, you need to add another 1 GB to 2 GB of memory just for the operating system. In addition, because SQL Server is memory starved, it is suggested that you add at least 2 GB just for SQL Server use. You also need to add a total of at least 4 GB of memory to solve memory bottlenecks. The processor subsystem is a bottleneck as well. Processors are frequently a bottleneck with regular peaks at 100%. Interrupt percentages and Privileged time percentages indicate optimal operating system and driver efficiency. For optimal performance, you should offload some workload from the server or upgrade the server with faster processors or replace the server. When you can address disk and memory bottlenecks, processor utilization will significantly increase over the current 43% average. The network subsystem is considered a bottleneck. While not severe, the network is often running at maximum receive rates. Resolving other bottlenecks within the memory, disk, and processor configuration results in greater network throughput demand and potentially significant network bottlenecks. You should explore multiple 100 Mbps adapter teaming or upgrading the server interface to run at 1 Gbps. Windows 2003 has an improved network stack capable of sustaining about 90% to 95% utilization compared to 75% to 80% for Windows 2000. An alternative would be to upgrade to Windows 2003 to obtain a slight improvement in usable network throughput. The disk subsystem is definitely acting as a bottleneck because of 8 KB stripe sizes and frequent 256 KB to 512 KB I/O to log and database files. For the C: drive, long write latency are associated with large and frequent paging writes. A default 8 KB stripe size translates to eight physical disk I/Os for the frequent 64 KB writes and would partly explain frequent long disk latency for writes. Analysis shows that about half of the write traffic is paging traffic. (The remainder is Perfmon logging.) Formatting the C: drive array using a 64 KB stripe size will likely improve disks array performance.<br />
<br />
Chapter 21. Case studies<br />
<br />
721<br />
<br />
Another analyzed drive is the D: drive. Analysis shows that peak latencies of 4 seconds occur when peak I/O requests are queued to LUN D:. Heavy load on the D: drive, especially because the stripe size is set to 8 KB (default), are noticed. As well, long latency is associated with very large 512 KB I/O to the D: disk. You should enable read/write cache when using RAID-5 and format the array with 64 KB stripe size. Because of very large writes, the array needs additional disks to absorb the heavy load. Alternatively, consider moving tempDB to another LUN because that is the sources of most read traffic and is about half the load. The final disk that was analyzed was the V: drive. Analysis showed moderate to heavy I/O load on the V: drive array coupled with very large writes causing long latency. The system is disk bottlenecked on V: due to very large I/O sizes and high I/O rates. You should upgrade the stripe size to 64 KB and increase the number of drives to 12 or more for this array.<br />
<br />
21.3 File servers hang for several seconds The configuration for this case study uses multiple servers, which consist of both file servers and application servers. The issue of file servers that hang for several seconds implies a long wait time for users who are based in Web-browsers. To help determine the problem and to solve it, the case study monitors the file server’s subsystems. The file server is a 4-way Xeon. Again, to gain a complete picture of what is happening on the file server’s subsystems, we need to monitor every element: memory, processors, network, and disks. Each component is related to the others, and a bottleneck issue on one component can impact the others. However, sometimes the actual bottleneck is not that obvious.<br />
<br />
722<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.3.1 Memory analysis We begin the scenario by analyzing the memory subsystem. Figure 21-11 shows that free memory is about 2.6 GB to 3.0 GB. There is no significant paging, and the cache is stable. Therefore, we can say there is no configuration issues, and this memory configuration is suitable for the current workload.<br />
<br />
Figure 21-12 Memory analysis<br />
<br />
Chapter 21. Case studies<br />
<br />
723<br />
<br />
21.3.2 Processor analysis Figure 21-13 shows the analysis of the processor subsystem. This analysis is interesting because it shows when the server hangs. The average processor utilization is only 15%. However, the analysis shows huge peaks at 99.9% when the server hangs. The deeper analysis shows that this processor utilization is essentially kernel time (meaning the operating system). From this analysis, you can determine that upgrading the processor to a faster one will not solve the problem, because the processor is busy an average of 15% and hang events would not be solved.<br />
<br />
Figure 21-13 Processor analysis shows hang events on peak utilization<br />
<br />
724<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.3.3 Network analysis The network analysis also shows hang events. The network subsystem appears to be under utilized and, therefore, is not causing a bottleneck. However, as shown on Figure 21-14, there is a peak throughput correlated to the spike in CPU utilization. The network average throughput is about 64 Mbps, and there is an average of 15 500 packets sent or received every second.<br />
<br />
Figure 21-14 Network analysis shows hang events<br />
<br />
The analysis of this subsystem shows that it is healthy. The LAN traffic is correlated with the CPU spike, which proves it is in response to a greater request in file servers load. The servers are then responding to the increased load from application server.<br />
<br />
Chapter 21. Case studies<br />
<br />
725<br />
<br />
21.3.4 Disks analysis of the V: drive The V: drive is used to store the data from the file server. Figure 21-15 shows the average read latency is around 3 ms, which is acceptable. Some minor spikes can occur but never above 20 ms so can be considered negligible. Moreover, the second graph shows an average size for read accesses around 4 KB.<br />
<br />
Figure 21-15 Disk read accesses do not indicate significant bottleneck<br />
<br />
726<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Alternatively, when monitoring the write accesses, Figure 21-16 shows an average write latency around 3 ms (which is good) and a peak at 630 ms. This peak is probably not due to the disk configuration and must be related to the CPU peak occurring when server hangs. With an average write operation of 5 KB, we can say that the stripe size for this drive should be 8 KB.<br />
<br />
Figure 21-16 V: drive write accesses analysis does not show bottlenecks<br />
<br />
Devices analysis did not show why the server hangs for a couple of seconds. Only CPU spikes are indicating there is a problem occurring, but the network, the memory, and the disk subsystems do not appear to be bottlenecks. Thus, at this point, we need to perform further analysis on a system level.<br />
<br />
Chapter 21. Case studies<br />
<br />
727<br />
<br />
21.3.5 System-level analysis To find the cause of the hang event, we will keep the counter demonstrating the event (in our case, the processor time percentage) and add progressively the counters for each object to be examined. These objects include System, Server, and Process. You can monitor many counters for each object. For this scenario, the first counter that we monitor is the File Control Operations Per Second on the system object (as shown in Figure 21-17, the white line). On average, the server is performing about 1300 file control operations per second. But during the CPU spike, the application server produces a storm of 10,000 file control operations per second. This spike consumes the CPU to nearly 100% utilization and appears to be a root cause of the performance issue.<br />
<br />
Figure 21-17 Analyzing file operation counters<br />
<br />
Another counter monitored is the File Open counter on server object (Figure 21-17, the blue line) Here again, the number of files that are open correlates with the CPU utilization spike. This correlation confirms that the server is opening a large number of files and drives the CPU utilization to nearly 100%. During normal operation, the server has only about 25 files open. However, when the peak occurs, the application servers request that the file server open as many as 200 files. To buffer the file control structures, the server allocates Pool Nonpaged Bytes (Figure 21-17, the grey line).<br />
<br />
728<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Pool Nonpaged Bytes is memory that is non-pageable and allocated to store file control structures (directory and file pointers). Figure 21-17 shows that, at the peak moment, 8 MB are allocated to nonpage pool, which is not excessive. Because less files are open, less memory is used and that is why there is the progressive decrease of bytes allocated to nonpage pool. At the time of the peak, the amount of data that is sent or received by the file server drops to zero because the CPU is 100% utilized processing file control operations. Because file control operations spike to over 10 000 control operations per second, we can confirm that this is the root cause for high CPU utilization and, therefore, for the application hang. A solution is to increase the number of file servers so that the opened files are spread over multiple nodes.<br />
<br />
21.4 Database server This case study analyzes a heavily loaded database server and suggests methods to improve its performance. The server in this case study has the following configuration: xSeries 340 with two 733 MHz Pentium III processors IBM ServeRAID-4H controller, connected to an external IBM EXP300 disk enclosure with fourteen 18 GB hard drives Let us take a look at the performance statistics of the different server subsystems.<br />
<br />
Chapter 21. Case studies<br />
<br />
729<br />
<br />
21.4.1 CPU subsystem Because this is a dual processor machine, we first check whether the workload is evenly balanced across both CPUs. Uneven CPU balancing is often a problem with existing applications that do not account for SMP machines. Figure 21-18 shows that both CPUs are utilized equally well. While minimum and maximum utilization varies greatly, the average load in the CPUs is 42% to 44%. This usage indicates that the CPU subsystem is not the bottleneck for this system. A value below 70% is acceptable and leaves enough headroom for occasional spikes.<br />
<br />
Figure 21-18 CPU utilization<br />
<br />
730<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.4.2 Memory subsystem To determine if the memory subsystem is a bottleneck, we determine if the system is paging a lot of data to the disk, which might be a symptom of a system running low on memory. To give an overall impression of memory utilization, we also take a look at the Memory:Available Bytes counter. Figure 21-19 shows that there is plenty of system memory available, yet the system is performing a substantial amount of page reads. (Note that the Page Reads/sec counter is shown on a 0.10 scale.) We need to determine why this bottleneck is occurring.<br />
<br />
Figure 21-19 Memory utilization<br />
<br />
Recall what the Page Reads/sec and Page Write/sec counters actually represent: Page Reads/sec is the number of times the disk was read to resolve hard page faults. Hard page faults occur when a process requires data that is not in its working set or elsewhere in physical memory and must be retrieved from disk. This counter was designed as a primary indicator of the kinds of faults that cause system-wide delays. It includes reads to satisfy faults in the file system cache (usually requested by applications) and in non-cached mapped memory files. This counter counts numbers of disk read operations, without regard to the number of pages retrieved by each operation. Page Writes/sec is the number of times pages were written to disk to free up space in physical memory. Pages are written to disk only if they are changed<br />
<br />
Chapter 21. Case studies<br />
<br />
731<br />
<br />
while in physical memory, so they are likely to hold data, not code. This counter counts write operations, without regard to the number of pages written in each operation. The high number of page reads (a value higher than 50 or so per second would normally indicate a memory problem) suggests that the system is running low on memory and is paging heavily. However, when we look at the page writes for this scenario, they are constantly zero. This pattern is not the pattern that is an indicator of a low memory problem. Instead, this behavior suggests that the paging activity is done by the application itself. The use of memory mapped files in an application usually accounts for this. See the following Knowledge Base entry for information about memory mapped files: http://support.microsoft.com/default.aspx?scid=kb;EN-US;q139609 Because this issue appears to be with the application itself, you would normally talk with the software vendor that supplied the application. If you cannot talk to the vendor, make sure the paging device is a fast array (ideally, many 15 K RPM disks in a RAID-10 array) with a 32 KB or 64 KB stripe size, because this is larger than the average size of pagefile accesses. The average pages read per second value of 235 keeps two to three 10 K RPM disk drives busy.<br />
<br />
732<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.4.3 Disk subsystem Next, let us take a look at the disk array to determine whether the disk subsystem is a bottleneck (Figure 21-20).<br />
<br />
Figure 21-20 Disk counters for Disk 0<br />
<br />
From our analysis, we can determine the following key disk metrics: Stripe size Average disk bytes/read are 5267 and the maximum is 6498 bytes. So, as far as reads are concerned, the stripe size at 8 KB is acceptable because this is larger than both maximum read and write sizes. Looking at write traffic, the Average Disk bytes/write are 5943 but the maximum is 38 297 bytes. The data shows a regular write size above 16 KB, so a 32 KB stripe size would be best for this array.<br />
<br />
Chapter 21. Case studies<br />
<br />
733<br />
<br />
I/O rates Disk transfers/sec are 626 average and 1093 at peak. The ServeRAID-4H controller can handle over 2500 random I/Os per second for RAID-5 arrays, so the controller is not under stress. So far, the only problem we have encountered is that the array uses a non-optimal stripe size. Let us move on and examine the Average disk sec/write counter, which describes the amount of time a write operation takes on a given disk drive. Disk latency Figure 21-21 shows the writes are experiencing the most delay, but reads, which are not shown here for clarity, are not much better. Average disk sec/write are 23 ms while the maximum is 212 ms. The maximum is an extremely long period because we want to keep disk operations below 20 ms to 25 ms.<br />
<br />
Figure 21-21 Average disk sec/write counter<br />
<br />
We first checked whether the ServeRAID controller was configured for write through, but it was verified that the controller is configured for write back. Write back is the best setting for this configuration and workload. However, given write back with an 8 KB stripe size, an I/O request forces two physical<br />
<br />
734<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
disk I/Os to two different disks each time a 16 KB request is issued, and for the few cases where the request is greater than 24 KB, three disk I/Os occur. We estimated by looking at the distribution of the Average disk bytes/write counter in Figure 21-20 on page 733, that about 2.3 to 2.5 physical disk I/Os occur each time the software requests a logical I/O. Adjusting to a larger stripe size reduces this to something closer to one disk I/O per request. Looking at Figure 21-20 on page 733, it is clear the array is performing regularly over 800 I/Os per second. However, remember this is the logical I/O rate (the I/O rate requested by the application). Because the stripe size is incorrect, the disks are doing about 2.3 real I/Os for each application request. With this 8 KB stripe size, the disks are really doing about 2.3 * 626 or 1439 I/Os per array or 1439/13 = 110 I/Os per disk. This rate is driving the disks pretty hard and explains the long latency. So, when you fix the stripe size to 32 KB, the disk bottleneck will improve and throughput will increase. By switching to 32 KB, you effectively double disk throughput because each request generates, on average, one disk I/O (instead of 2.3 with the 8 KB stripe). System performance should improve about 50% to 60% when you change the stripe size to 32 KB. Rule of thumb: Each time you double disk I/O throughput, you get about a 50% improvement in system throughput. The disks are performing about 1400 I/Os per second (with higher server throughput) and the 13 disks will bottleneck before the server hits maximum performance (100% CPU). Number of disks, RPM, and RAID levels Given that the server CPUs are running at about 42% to 44%, you could get about double the current performance by adding more drives. In addition, you would need to add sufficient drives to bring the average I/O rate per disk down to 50 to 60 to reduce latency to around 10 ms to 15 ms, which would be ideal. So, at 1400 logical I/Os (and 1400 physical I/Os with a 32 KB stripe), the CPU utilization should be about 1.6 * 44% = 70%. If you want to improve performance even more so the server runs to 100% CPU utilization and reduce the latency so response time is faster, you have to add more disks or change to RAID-10. In general, RAID-10 has about 50% greater sustained throughput than the same number of disks configured using RAID-5, at least when ever random writes are a significant component of the workload.<br />
<br />
Chapter 21. Case studies<br />
<br />
735<br />
<br />
For example, at 1400 I/Os and 70% CPU utilization, you need to increase system performance by 30% to reach 100% CPU. Thus, you need 30/50*2*1400 = 1680 I/Os per second. For the server to run at 100% CPU utilization with acceptable response times, you must add drives so that each drive does no more than about 100 I/Os per second. This translates into 1680 I/Os / X disks = 100 I/Os/disk (max) = 17 RAID-5 disks. Thus, the optimal RAID-5 solution is to configure 17 disks total in a RAID-50 array. Ideally, you could move to 15K RPM disks and get a 25% boost in I/O rate per drive to reduce the number of disks to about 17/1.25 = 14 disks. Alternatively, you could go to RAID-10 to reduce the disks by half. Rule of thumb: With RAID-10, you can match the performance of a RAID-5 array with only half the number of disks. In summary, to run at 100% CPU utilization and have optimal response time, using RAID-50, the new array should have about 17 10 K RPM disks, or 14 15 K RPM drives in a RAID-10 array.<br />
<br />
736<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.5 ERP application server This case study, while not a production workload, shows another example of using Performance Monitor to examine key counters to detect bottlenecks. The server for this case study has the following configuration: Four 500 MHz Xeon processors Two RAID arrays: – One 4-disk array for operating system code and paging file, 8 KB stripe – One 4-disk array for data, 8 KB stripe size We examine the major subsystems using key Performance Monitor counters.<br />
<br />
21.5.1 CPU subsystem Figure 21-22 shows the percentage Processor time for the four processors.The CPUs peak at 40%. However, most of the time, CPU use is near zero. Closer examination of the four processor counters also shows that all four CPUs are occupied equally by the application, a sign that the application is tuned properly for SMP operation.<br />
<br />
Figure 21-22 Monitoring CPU utilization<br />
<br />
Chapter 21. Case studies<br />
<br />
737<br />
<br />
21.5.2 Disk subsystem A key counter to use when measuring disk performance is the average disk queue length, as shown in Figure 21-23. This disk is a 4-disk RAID array. Table 19-3 on page 669 shows that if the average disk queue length is consistently over 12 (at most three per disk), then there is a bottleneck. Because the queue length is almost always less than 12 (except for two spikes), we conclude that this disk is not a primary bottleneck. Similar analysis of the other RAID array (instance 1 in Figure 21-23) also confirms this theory.<br />
<br />
Disk queue length = 6 Disk queue length = 4<br />
<br />
Figure 21-23 Monitoring disk utilization<br />
<br />
738<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
We also examine the average bytes per transfer for both the paging disk and the data disk in Figure 21-24 and Figure 21-25, respectively.<br />
<br />
Figure 21-24 Monitoring I/O size: paging file<br />
<br />
We conclude from this analysis that because this disk includes only the operating system and the page file this I/O activity is most likely to be paging activity. This analysis might indicate a memory bottleneck. In Figure 21-24, the I/O operations are about 25 KB per I/O transfer. However, the array is configured with an 8 KB stripe size. After you add extra memory, you should re-examine the disk bytes/transfer counter. If the I/O size is still about 25 KB, we recommend that you set the stripe size of the page device to 32 KB.<br />
<br />
Chapter 21. Case studies<br />
<br />
739<br />
<br />
Figure 21-25 shows the average I/O transfer size in bytes for the data disk. Here the I/O size is between 20 KB and 64 KB. Because the disk array was configured with an 8 KB stripe size, there are up to eight I/O operations made to the disk array for each I/O request. We recommend that you re-create the array with a stripe size of 64 KB.<br />
<br />
Figure 21-25 Monitoring I/O size — data disk<br />
<br />
740<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
21.5.3 Memory subsystem To analyze the memory subsystem, we examine the memory page reads per second. This is the number of times the disk is read to retrieve pages of virtual memory necessary to resolve page faults, as shown in Figure 21-26. During the same time that I/O operations are occurring to the disks (the most probable period of production activity), memory pages are read between 35 and 44 pages per second. As shown in Table 19-2 on page 664, a sustained value of 5 page reads per second is considered a potential bottleneck. So, in this server, there is too little RAM installed in the server.<br />
<br />
Figure 21-26 Analyzing the memory subsystem<br />
<br />
Chapter 21. Case studies<br />
<br />
741<br />
<br />
21.5.4 Summary From these case studies, we found: <br />
<br />
The CPU is not a bottleneck. The application is using all four processors equally. The server has insufficient RAM installed. The lack of memory results in excessive paging to disk. The stripe sizes of the two RAID arrays are configured incorrectly.<br />
<br />
From our analysis, we were unable to examine the network subsystem, either because the counters were not recorded in the log or because the appropriate network monitor services were not enabled (see 19.5.1, “Finding network bottlenecks” on page 673 for details).<br />
<br />
742<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Part 6<br />
<br />
Part<br />
<br />
6<br />
<br />
Applications A server provides services to users. That service can be as straightforward as file and print or as complex as Web-based e-commerce with a back-end database and supply chain management. This part covers some of the more common server applications. In each chapter, we describe the key server hardware subsystems, how to tune the operating system, and the key tuning parameters for the application itself. The applications that we discuss in this part are: Chapter 22, “File and print servers” on page 745 Chapter 23, “Lotus Domino” on page 763 Chapter 24, “Microsoft Exchange Server” on page 831 Chapter 25, “IBM DB2 Universal Database” on page 847 Chapter 26, “Microsoft SQL Server” on page 861 Chapter 27, “Oracle” on page 923 Chapter 28, “Microsoft Windows Terminal Services and Citrix Presentation Server” on page 951 Chapter 29, “Microsoft Internet Information Services” on page 981 <br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
743<br />
<br />
744<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
22<br />
<br />
Chapter 22.<br />
<br />
File and print servers This chapter describes the unique performance characteristics of file servers and print servers. It refers to file server tuning for Windows Server 2003 and when using the Samba functionality of Linux with Windows clients. In this chapter, we discuss the follow topics: 22.1, “File servers” on page 746 22.2, “Print servers” on page 761<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
745<br />
<br />
22.1 File servers File serving is a very common use for IBM System x servers. The performance of how well a server delivers file services depends on a combination of the server hardware configuration and the operating system and its memory management and I/O subsystem. The four network subsystems need to be optimized to get the best file server performance. These are listed below in order of importance from most to least. Network The network subsystem is critical in ensuring that at a server can responsively service requests for files. This comes through a combination of using the right network interface and configuring it for the highest performance both within the operating system and at the switch it is connected to. See 19.5.1, “Finding network bottlenecks” on page 673 for bottleneck analysis of this subsystem. Memory Insufficient memory might limit the ability to cache files and thus cause more disk activity through paging, which slows down the system. You should have enough memory in your file server so that as little paging as possible to disk occurs. Avoid putting the pagefile on a RAID-5 array as this slows performance, and avoid placing multiple pagefiles on different partitions on the same physical drive(s). See 19.3.2, “Virtual memory system” on page 663 for bottleneck analysis of this subsystem. Disk Because the purpose of a file server is to supply files to the client, the server must initially read all data from a disk. Thus, the disk subsystem needs to be optimized. Adjust the allocation unit size according to the size of the clusters on the disk. If possible, keep the paging file, the operating system and the data on separate physical disk drives and use arrays of multiple disks. See 19.4.1, “Analyzing disk bottlenecks” on page 669 for bottleneck analysis of this subsystem. Processor In a file server, the processor subsystem is the least likely to cause a bottleneck.The CPU typically only becomes the bottleneck if the data is transferred from the network in small block sizes, as described in 10.2, “Factors affecting network controller performance” on page 242.<br />
<br />
746<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
22.1.1 The effect of server hardware on performance Figure 22-1 shows the performance characteristics of a typical file server. As the number of users in the network or the server workload increases, the performance of the network can be considered network intensive. The throughput increases until it reaches a limiting peak, and then it tapers off as the slower disk subsystem becomes apparent. The more transactions are processed, the more intensive the activity to disk are required. This activity can be partially offset by having a large amount of RAM in the server and using the optimal operating system settings for the file system cache. All the same, as client requests and network activity becomes heavier, the performance moves into the disk-intensive phase, which limits the server's performance.<br />
<br />
Peak memory intensive<br />
<br />
Increasingly network intensive<br />
<br />
Disk intensive<br />
<br />
Throughput or transactions per second<br />
<br />
High NIC utilization and high disk cache hit rate<br />
<br />
Poor disk cache hit rate high disk utilization<br />
<br />
Increasing number of users or server workload Figure 22-1 File server performance characteristics<br />
<br />
Chapter 22. File and print servers<br />
<br />
747<br />
<br />
Tests have shown that by using a faster network card, peak performance can be improved for lower loads, but throughput is restricted at higher loads by the disk subsystem, as shown in Figure 22-2 on page 748. To determine if your network is a performance bottleneck, examine the Performance counters listed in 19.5, “Network bottlenecks” on page 672.<br />
<br />
Throughput or transactions per second<br />
<br />
Benefit of adding a faster network adapter is maximized during high cache hit rate Benefit drops off as disk I/O rate becomes the bottleneck<br />
<br />
Increasing number of users or server workload<br />
<br />
Figure 22-2 The effect of adding a faster network card<br />
<br />
Using a faster disk subsystem (number of disks, mechanical speed, and disk cache), the peak throughput performance can be extended for higher server loads, as shown in Figure 22-3. The upgrading of both network and disk subsystems will provide cumulative improvements. To determine if your disk subsystem is a performance bottleneck, examine the Performance Monitor counters listed in 19.4, “Disk bottlenecks” on page 667.<br />
<br />
Server with faster disk subsystem<br />
<br />
Throughput or transactions per second<br />
<br />
Server with slower disk subsystem<br />
<br />
Increasing number of users or server workload<br />
<br />
Figure 22-3 The effect of adding a faster disk subsystem<br />
<br />
748<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
As mentioned previously, increasing the RAM and tuning the operating system can also help by increasing the size of the file system cache. Adding RAM also has the effect of extending the peak throughput for higher loads, as shown in Figure 22-4. To determine if your memory subsystem is a performance bottleneck, examine the Performance counters that are listed in 19.3, “Analyzing memory bottlenecks” on page 661.<br />
<br />
Server with additional memory<br />
<br />
Throughput or transactions per second<br />
<br />
Server with insufficient memory<br />
<br />
Increasing number of users or server workload Figure 22-4 The effect of adding memory<br />
<br />
22.1.2 Network subsystem The main purpose of any file and printer sharing is to provide resources across the network, therefore the network subsystem and configuration plays an important role. The rules for sizing and determining the network requirement on Samba can be applied to other file share protocols/applications. An easy formula to calculate the estimated bandwidth required is:<br />
<br />
number of clients x transfers per second x average file size (KB) = KBps used So, for example, if we have an office of 500 users and each user does 2 transactions a second on a file that is 25 KB, our bandwidth used would be approximately 25 MBps.<br />
<br />
Chapter 22. File and print servers<br />
<br />
749<br />
<br />
22.1.3 Disk subsystem Because the major function of the file server involves disk I/O, serious consideration should be given to this area of the server. We recommend the following disk configuration: Use RAID-1/1E, RAID-1+0, RAID-5/5E, or RAID-50 for hard drive fault tolerance. RAID-1/1E and RAID-1+0 will give you better performance, but RAID-5 will give you more usable space for the same number of disks. Place the operating system code, swap/pagefile partition and application code on drive(s) separate from the shared data. The operating system code should not be accessed very often when the server has booted, and provided the server has enough memory, the swap/pagefile partition should not be accessed very often either. For ServeRAID-based systems, enable write-back cache (with the battery-backed cache, this is the default; without it, the default is write-through). This allows all writes to disk to be stored in a cache, which permits the system to continue operations without waiting for confirmation that data has been written to the hard disk. Install the latest disk controller BIOS, firmware, and drivers (e.g. ServeRAID or Fibre Channel). Use the fastest drives possible. (For more information see Table 20-2 on page 699.)<br />
<br />
22.1.4 Tuning Windows Server 2003 Windows Server 2003 is well optimized out-of-the-box for general file serving purposes. Under some conditions, you can improve the performance of your file server by changing various operating system settings. Tip: You should also refer to Chapter 11, “Microsoft Windows Server” on page 295 for more complete details about tuning this operating system. The following parameters are worth investigating in any given file server implementation: <br />
<br />
750<br />
<br />
Foreground application boost How the system cache is used Page file size setting Registry settings Volume shadow-copy services<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Application responsiveness As discussed in 11.2, “Windows Server 2003, 64-bit (x64) Editions” on page 298, you can configure how Windows Server 2003 responds to foreground and background applications as part of their multitasking capability. Optimizing performance for background services is much better suited to the requirements of file server. You can set the application response to background services by following these steps: 1. 2. 3. 4. 5.<br />
<br />
In the Control Panel, open System. Select the Advanced tab. In the Performance box, click Settings. Select the Advanced tab. The window shown in Figure 22-5 opens. Optimize performance by clicking Background services. Set the application response to Background services to improve background application performance.<br />
<br />
Figure 22-5 Disabling boost of foreground applications in Windows Server 2003<br />
<br />
System cache As described in 11.6, “File system cache” on page 309, you can control how Windows Server 2003 prioritizes the size given to the system cache. Follow these steps: 1. 2. 3. 4. 5.<br />
<br />
Open the properties window for your network interface. Select the General Tab and click Properties. Select File and Print Sharing for Microsoft Networks. Click Properties. The window shown in Figure 22-6 on page 752 opens. Select Maximize data throughput for file sharing. Click OK.<br />
<br />
This preferred setting allocates more memory to the file system cache and less for network connections to your server. Selecting this option instructs the operating system to give the working set of the file system cache a higher priority for memory allocation than the working sets of applications. It yields the best performance in a file server environment that is not running other applications locally.<br />
<br />
Chapter 22. File and print servers<br />
<br />
751<br />
<br />
Set this option for file servers<br />
<br />
Figure 22-6 Setting Windows Server 2003 to a file serving role<br />
<br />
Note: As described in 11.6, “File system cache” on page 309, the system cache is limited to 960 MB in 32-bit (x86) versions of Windows Server 2003. This limit is also discussed in knowledge base entry KB837331: http://support.microsoft.com/default.aspx?scid=kb;en-us;837331<br />
<br />
Virtual memory Windows uses a virtual memory system that uses a combination of physical memory (RAM) and hard disk space (the paging file) to temporarily hold recently or frequently accessed data that is not currently being executed by the system CPUs. Allowing memory to page to the disks on the server is a technique used to reduce the amount of physical RAM that a server must include in order to run the operating system. For optimal performance, a server should be configured with sufficient RAM to reduce or prevent sustained paging from occurring. All servers will occasionally page during the boot process and while applications start up. With the price of RAM every decreasing, it is a very straightforward and relatively inexpensive way to improve file server performance considerably.<br />
<br />
752<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To reduce the chance of having insufficient virtual page space, which can cause the server to slow or become unresponsive, you should set the virtual memory size appropriately. Use the value recommended by the operating system. 1. In Control Panel, open System. 2. Select the Advanced tab. 3. In the Performance box, click Settings. 4. Click Properties. 5. In the Virtual Memory box, click Change. The window shown in Figure 22-7 opens. 6. Enter in new values for Initial Size and Maximum Size and click OK.<br />
<br />
Create paging files on multiple drives to improve paging performance<br />
<br />
Figure 22-7 Page file size and distribution in Windows 2003<br />
<br />
For best performance, we recommend that you set the total page file to be twice the size of the amount of physical RAM installed in the server. We have often seen examples of severely modified page file configurations distributed over many varied operating system volumes that are actually just on the same underlying physical disks. The situation is made worse when these volumes are spanned across multiple Windows dynamic disk partitions. File server performance is best achieved if the page file is made of multiple segments<br />
<br />
Chapter 22. File and print servers<br />
<br />
753<br />
<br />
spread across multiple physical disks (or arrays of disks) of the same speed and size. This achieves nothing but further complexity and likely, degraded system performance. Of course, the best tuned page file in the world is still no substitute for adequate physical memory. A busy file server should always be monitored for page file activity using the appropriate memory and paging file counters in System Monitor and tuned appropriately. Minimums and maximums: When specifying a page file size, increase the minimum value to equal the maximum value. If these are set to different values, when the page file needs to grow or shrink, it will slow down your server in the process. By setting both values as equal, you will expend some disk space but the system does not need to spend time sizing the page file.<br />
<br />
Registry parameters1 These are some registry parameters that could affect the performance of file servers. These include: PagedPoolSize HKLM\System\CurrentControlSet\Control\SessionManager\MemoryManagement\ File cache space and paged pool space share a common area in system virtual address. Limiting the paged pool allows for a larger system cache, which causes more content to be cached and allows faster serving of files. The default value for this registry key is 0x0. The range goes from 0x0 to 0xFFFFFF. If you set the value to 0x0, the system calculates an optimal value for this entry based on the amount of physical memory in the computer, and it adjusts this value if the amount of memory changes. If you set a range from 0x1 to 0x20000000, the system creates a paged pool of the specified size. If you set the value to 0xFFFFFFFF, the system calculates the maximum paged pool allowed for the system in this configuration. For a file server the value recommended is 0xB71B000 (192000000). Setting this value provides the system with a large virtual address space, expandable to up to 960 MB. Note that a corresponding entry of zero (0) is required in the SystemPages registry value for this to take optimal effect. This value is found within the same MemoryManagement registry key.<br />
<br />
1<br />
<br />
754<br />
<br />
Reproduced with permission from Microsoft Corporation from Performance Tuning Guidelines for Windows Server 2003, which is available online at: http://www.microsoft.com/windowsserver2003/evaluation/performance/tuning.mspx<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
NtfsDisable8dot3NameCreation HKLM\System\CurrentControlSet\Control\FileSystem\ This parameter determines whether NTFS generates a short name in the 8.3 (DOS) naming convention for long file names and for file names that include characters from the extended character set. Setting the value to 0 (this is the default) enables applications that cannot process long file names and computers that use different code pages, supporting character sets and keyboard layouts for different countries or regions. Setting the value to 1 will increase the file server performance, as it avoids the short-name attribute creation for the file. The recommendation for a file server is 1, but you have to be careful to use this value, because applications and client operating systems that cannot process long file names, and computers that use different code pages, might not be able to find the files. On contemporary systems, this is unlikely to present any issues. DisableLastAccess HKLM\System\CurrentControlSet\Control\FileSystem\ By default, this registry key is not created. If you have an NTFS volume with a large number of folders or files, and you have an application locally or remotely which accesses many of these files, the I/O bandwidth used to update the Last Access Time property can be a significant percentage of the overall I/O bandwidth. To increase the speed of access to a folder or file, you can set DisableLastAccess to 1 to disable updating the Last Access Time. After you use this command and restart the server, the Last Access Time is no longer updated. If you create a new file, the Last Access Time remains the same as the File Creation Time. The result is faster hard disk file read-access. Note that in some high-security environments where file system accessing and auditing is required, disabling this Last Access Time value might not be permitted. NumTcbTablePartitions HKLM\system\CurrentControlSet\Services\Tcpip\Parameters\ By default this key is not created. This parameter controls the number of TCB table partitions. The TCB table can be partitioned to improve scalability on multiprocessor systems by reducing contention on the TCB table.<br />
<br />
Chapter 22. File and print servers<br />
<br />
755<br />
<br />
The value should be a power of two, that is, 2, 4, 8, 16, 32, and so on. On multiprocessor systems, change the number of partitions to four times the number of processors in the server. TcpAckFrequency TcpAckFrequency is a parameter that sets the number of TCP acknowledgments (ACKs) that will be outstanding before the delayed ACK timer is ignored. This parameter is only for Windows Server 2003 and Windows XP. For previous Windows versions, use the parameter TCPDelAckTicks. The recommended setting for TcpAckFrequency is between one-third and one-half of TcpWindowSize. For Gigabit cards: HKLM\system\CurrentControlSet\Services\Tcpip\Parameters\Interfaces For each Gigabit (1000 Mbps) adapter, add: TcpAckFrequency (REG_DWORD) = 13 (decimal) For each Fast Ethernet (100 Mbps) adapter, add: TcpAckFrequency (REG_DWORD) = 8(decimal) By default, this entry is not in the registry. With this setting, If only “acking” (acknowledging) data and not any control packets, an “ack” occurs once every 8 or 13 packets, instead of the default of two. This helps reduce packet processing costs for the Network Stack, in the case of large writes (uploads) from the client into the server. For Fast Ethernet cards: HKLM\system\CurrentControlSet\Services\Tcpip\Parameters\Interfaces For each Fast Ethernet adapter, add: TcpAckFrequency (REG_DWORD) = 5 (decimal)<br />
<br />
Tuning NTFS For Windows Server 2003-based file servers, NTFS is the recommended file system for all data and system partitions. NTFS is a reliable, journalling, performance optimized file system which also offers file-level security including permissions, auditing, and encryption. While FAT and FAT32 are generally faster than NTFS for small partitions (up to say a few GB in size), you should not use these on servers, because they are more prone to data corruption, especially on large volumes.<br />
<br />
756<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
There are two NTFS specific settings that can affect performance: Allocation unit (cluster) size Similar to the stripe-size of RAID controllers, the NTFS cluster (or allocation unit) size should be set to correspond with the actual usage patterns of the server. Matching the cluster size with the type of files stored will allow for greater disk throughput, because the server can read an entire file with one disk read if the whole file can be stored in one allocation unit. For files that are larger than one allocation unit, the number of reads is minimized if a large cluster size is used. On the other hand, using too large a cluster size will waste disk space, because clusters will only be partially filled with small files. For file servers, a value larger than the default 4096 bytes is typically recommended. A crucial tip when creating data volumes is ensuring the corresponding physical drive array and logical drives are configured using the same stripe size (controller) and cluster size (volume). Having the hardware managed stripe size match the operating system cluster size will allow the file system to perform at its best. To decide which size to use, it might be appropriate to generate statistics on the size of the files that you expect the server to host. If most files are small, such as typical or application specific files, a smaller cluster size of 16 KB or 32 KB might be appropriate. If you are storing large files like graphics, video or sound files, use a larger cluster size such as 64 KB. The cluster size can only be set when formatting a volume, so you cannot change it on existing volumes that already have data on them unless you want to move the data to another location, format the drive and then transfer the data back. You can set the cluster size using both the command line and GUI version of the format command. To specify the size at the command line, use the /A switch of the format command. For example, to format an NTFS volume with a cluster size of 32 KB, you would issue the following command: format x: /A:32K /fs:ntfs To format an NTFS volume using 32 KB clusters from the Windows GUI, use the Disk Management interface found within the Computer Management MMC and right-click the desired volume and select Format from the drop down menu. Specify the preferred allocation unit size, label the volume and click OK as shown in Figure 22-8.<br />
<br />
Chapter 22. File and print servers<br />
<br />
757<br />
<br />
Figure 22-8 Specifying an allocation unit size<br />
<br />
Note that using the format function in the native Windows Explorer interface will only allow a maximum cluster size of 4096 bytes, so to achieve larger allocation units, use the command line or the Disk Management tool. Note too that formatting volumes with cluster sizes of greater than 4096 bytes disables NTFS compression capability for that volume and also disable the functionality of the native Windows disk defragmentation tool. NTFS log file size Much like a database system, NTFS uses a transaction log or journal to temporarily store write data before it is committed to disk. On heavily used systems such as large file servers, the log file can fill up. NTFS will then expand the file to account for this. During the expansion, the volume that uses the log is locked and cannot be written to. You can expand the log file size permanently to prevent expansion taking place during normal system operation. To increase the NTFS log file to 64 MB, a recommend value for large file servers, issue the following command, where x is the volume to be modified: chkdsk x: /L:65536<br />
<br />
NTFS compression Because today the cost of drives is not expensive relatively, worrying about disk space is a questionable economy. NTFS compression, especially on servers, should be avoided. The compression process can impact CPU activity and slow down the disk operations. Sizing disks appropriately at server specification time and subsequent monitoring and capacity planning techniques and are far better as preventative measures than enabling NTFS compression as a method to regain lost disk space.<br />
<br />
758<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
22.1.5 Tuning Linux: Samba Linux also provides a server message block (SMB) server service called Samba. The Samba server acts as a file server for Windows clients. It can act as a stand-alone server or domain controller. Samba uses TCP/IP to communicate with the client. We now discuss some of the main settings to optimize the performance of a Samba server. All these settings can be found in the /etc/samba/smb.conf file.<br />
<br />
oplocks This option lets SMB clients cache files locally. If a server grants an oplock (opportunistic lock), then the client is free to assume that it is the only one accessing the file and it will aggressively cache file data. This can provide big performance benefits. oplocks is enabled by default and should only be disabled when dealing with unreliable network connections. It is also possible to enable or disable oplocks on a by-share basis. oplocks = yes<br />
<br />
level2 oplock level2 oplocks (read only oplock) allows Windows clients that have an oplock on a file to downgrade from read/write oplock to read-only oplock. This will happen if another client accesses the same file on the Samba server. This allows all clients accessing the files to cache the file for read-ahead only, increasing performance for many accesses of files that are not commonly written (such as application files). When one of the clients which have a read-only oplock writes to the file, all clients are notified and told to break their oplocks to “none” and delete any read-ahead caches. It is recommended that this parameter be turned on to speed access to shared executables. The default value for level2 oplocks is enabled as follows: level2 oplocks = yes<br />
<br />
Memory For each user who connects to a Samba server a new smbd daemon is spawned and uses on average approximately 1-2 MB of memory. Based on this one can do some rough calculations on how much memory will be used by the Samba application.<br />
<br />
Chapter 22. File and print servers<br />
<br />
759<br />
<br />
Samba specific optimizations The parameters for tuning Samba are located within the Samba configuration file /etc/samba/smb.conf. There are a number of socket options that can greatly affect the performance of a PCT-based server like Samba. Getting the socket options right can make a big difference to the performance of the Samba server. The correct settings are very dependent in the network environment. Socket options are targeted specify at the TCP networking layer. Within Samba, these options can be specified on the command with the -o option or with the sock options parameter within the Samba configuration file. Red Hat Enterprise Linux and SUSE LINUX Enterprise Server have already set these for good performance however ensure that the following options are set are set for a local area network install of Samba: socket options = IPTOS_LOWDELAY TCP_NODELAY If you are installing Samba into a wide area network try the following options socket options = IPTOS_THROUGHPUT TCP_NODELAY The TCP_NODELAY seems to have the biggest impact for most networks. Tests have shown that adding the following can increase the read performance of a samba drive by as much as 50%: socket options = TCP_NODELAY For more information about the available socket options parameters, consult the smb.conf man page. Another area that can dramatically impact performance is the log level option. Normally, if you want to debug a problem, the log will give you good information about what happened on the server. The amount of information stored in the log depends on the value of the setting log level. The default is 0, which means that no logging is enabled. The higher this value, the more detailed the logged information will be. If this option is set higher than 2 on a production machine will suffer from degraded performance due to excessive logging. To maximize performance, make sure you have the following entry in smb.conf: log level = 0 Note: Log level is also called debug level. The following setting gives you the same results: debug level = 0<br />
<br />
760<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
In a RAID environment you will want to set the write cache size to be equal RAID stripe size, the write cache parameter is used for controlling when certain information is saved to disk. For example if you have a stripe size of 32K to would set it to 32768 as samba reads the value in bytes. write cache size = 32768 Other options that might improve performance but should be tested in your environment are: read raw = no write raw = no<br />
<br />
22.2 Print servers The main goal for a print server is to share access to print resources over the network. A well planned print environment is the key for a good performance on print services. In many situations, the print server function in an office is shared with the file server. In large companies with many print devices however, it makes sense to separate this function onto a separate server altogether, to improve performance and to ensure printing services do not impact the performance of the file server and vice versa. Tuning techniques for print servers will typically build upon the best practices for file servers outlined in the previous section of this chapter. Moving the printer spool area to another volume (and underlying disk or array) other than the default (typically where the operating system resides) is always recommended. This will ensure printing does not interfere with operating system disk activities and will optimize performance. In a Windows environment, we also recommend that you integrate the print servers with Active Directory. By doing so, you will have a central point to view print devices, drivers, availability and access permissions and security control. Using Active Directory, users can easily search and locate printers with features that meet certain criteria such as location, color support, capacity etc. Using the latest print drivers is an important step in ensuring print server performance and stability. It only takes a single faulty print driver with a memory leak to render a large print server hosting dozens or even hundreds of queues inoperable. Always check with the manufacturers Web site to ensure you are using the latest print drivers for any print device in your enterprise.<br />
<br />
Chapter 22. File and print servers<br />
<br />
761<br />
<br />
Another best practice technique is to create multiple logical printers to a same device so you can schedule different work times. For example you can direct a large document to a logical printer that has been specified to only function between 12:00 a.m. to 6:00 a.m. Sending documents to appropriately scheduled logical printers is a good way to improve print server performance throughout the day. Print servers do not typically suffer the same subsystem taxing that more traditional server functions might. Processor, memory and disk subsystems are not normally constrained on print servers given the sporadic nature of their client requests. The network system can occasionally become a bottleneck on large print servers where many clients are connected simultaneously, even if they are not printing all the time. Typically though a highly configured print server is capable of supporting thousands of printers and even more users without suffering any level of performance anxiety. As a final note, the new Print Management Console in Windows Server 2003 Release 2 (R2) offers a considerably easier interface to centrally manage the printing environment of a company that has distributed print servers, including those in remote offices. Print server administrators can more easily monitor queue status, be notified of and respond to printing errors, deploy printer connections and automate the install of remote printers. The Print Management Console is an MMC snap-in that allows customized views, for example, all the printers in a particular state or all those with particular features. These features combined will facilitate a more manageable and more optimized printing environment for both users and administrators.<br />
<br />
762<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
23<br />
<br />
Chapter 23.<br />
<br />
Lotus Domino Lotus Domino is a popular application and messaging system that enables a broad range of secure, interactive business solutions for the Internet and intranets. The Lotus Domino server and Lotus Notes client are powerful tools for communication, collaboration, and sharing information. As with other application servers, careful planning, maintenance, and tuning processes are essential in systems administration. This chapter discusses the tuning concepts and procedures of Lotus Domino server running on Windows Server 2003. For information about Domino performance beyond what we cover in this chapter, see the following Web sites: http://www.lotus.com/performance http://www.notesbench.org You can also find more information in the paper Domino 7 Performance Tuning Best Practices to Get the Most Out of Your Domino Infrastructure, REDP-4182, which is available at: http://www.redbooks.ibm.com/abstracts/redp4182.html Lotus Domino server can act as a mail, Web, application, and database server that handles mail routing and storage as well as database and Web requests. The important subsystems for Domino server are disk, memory, CPU, and network.<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
763<br />
<br />
23.1 Performance tuning outside of Domino In order for Domino to perform its best, other areas outside of Domino must be performing their best as well. Domino relies on the operating system that is hosting the server, the hardware that is running the operating system, and the network that is connecting it all to perform well. If any of these components is configured poorly or is underpowered, then Domino’s performance is impacted negatively. This section does not provide detailed configuration and tuning information for any one type of network, server hardware, or operating system. Instead, it touches on general tuning concepts that you should consider and includes some operating system specific examples to determine whether tuning might be necessary.<br />
<br />
23.1.1 Network performance The network is the backbone for the entire Domino infrastructure. The connection between a client (Notes, Web, and so forth) and a Domino server can only be as fast as the network allows. Therefore, if the network does not have sufficient bandwidth to accommodate the load, then every network related activity is impacted negatively, including Domino. (You can find detailed information about the network subsystem and its performance in Chapter 10, “Network subsystem” on page 235.)<br />
<br />
23.1.2 Platform architecture and system performance As is the case with network performance, the hardware platform—the physical machine that is running the software—can dictate whether Domino performs at an acceptable level. The system’s architecture, CPUs, RAM, and disk subsystem all play a part in the performance of the system and, therefore, Domino’s performance as well. If any one of these components is under performing, then the system as a whole performs poorly. Note: The hardware for both servers and clients has an impact on the perceived performance of a Domino server. So, both need to be configured correctly.<br />
<br />
CPU In most cases, the number of CPUs or cores on a system has a more significant impact on a server’s performance than the actual speed of the CPUs. Loosely speaking, each CPU can process a separate instruction from other CPUs on the<br />
<br />
764<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
system simultaneously instead of having to wait for a single CPU. You can find more information in 4.3, “Processor performance” on page 66. Many Domino functions are processor-intensive. Tasks such as routing messages, indexing databases, searching databases, and creating HTML pages for Web clients dynamically all put heavy demands on the system’s CPU. Domino takes advantage of multiple processors by allocating the replicator and router tasks to different CPUs, thereby spreading the load. Domino is a highly scalable application that can take advantage of the benefits of SMP. It does so automatically. You do not need to tune or to configure your hardware for Domino to take advantage of the additional CPUs.<br />
<br />
23.1.3 Memory Different platforms have different limitations on the amount of RAM that they can support directly. For example, 32-bit platforms such as the x86 architecture are only able to access 4 GB of physical RAM directly, while 64-bit architectures are theoretically able to access as much as 16 Exabytes. For more information, see Chapter 8, “Memory subsystem” on page 129. The amount of RAM that is available to a given process (including Domino) is further restricted by operating systems as well. Operating systems can combine the physical RAM with additional hard drive space to create virtual memory. This virtual memory is where allocations of memory to processes occur, and it is up to the operating system to distribute the memory to the applications. You can find more information about this in 23.2.1, “Operating system memory allocation” on page 768.<br />
<br />
23.1.4 Disk subsystem Hard drive performance can be one of the most often neglected components in a system, yet increasing performance here can yield significant improvements for the entire system as described in Chapter 9, “Disk subsystem” on page 169. The disk subsystem is the slowest subsystem in a modern server. So, with disk-intensive applications such as Lotus Domino, it is also often the biggest reason for bottlenecks. Before outlining some disk subsystem best practices, it might be useful to mention a couple of ways that Domino uses disks: Domino I/O streams We recommend using Logical Raid Units and balancing the Domino data files over these units. If the logical unit consists of 16 disk drives, for example, then<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
765<br />
<br />
the queue length should average 16 to 32, which translates to 1.5 to 2 per disk. For more information, see Chapter 9, “Disk subsystem” on page 169. Domino’s use of the page file Lotus Domino uses a process known a memory mapping to share memory pages between processes. Memory mapping allows the application to write physical areas of the memory to the paging file on disk and then read them back again. For example, if process one has a working set of data in memory and process two needs access to the same data, the application can write the page directly to the paging file and then read it back into a memory area available to process two, as shown in Figure 23-1.<br />
<br />
Disk drive<br />
<br />
Executable image Executable Image image<br />
<br />
Process 1<br />
<br />
System pagefile<br />
<br />
Process 2<br />
<br />
2 GB<br />
<br />
0<br />
<br />
2 GB Code pages<br />
<br />
Code pages<br />
<br />
Data pages<br />
<br />
Shared data<br />
<br />
Shared data<br />
<br />
Data pages 0<br />
<br />
Figure 23-1 Sharing memory using memory mapping<br />
<br />
Applications use the paging file rather than simply writing the data to and from a file on the data drive because of speed. The entire process of writing to the paging file is optimized to make it as fast as possible. Thus, no matter how large your physical memory, Domino always uses the system paging file to share some memory areas.<br />
<br />
Best practices: disk subsystem The following are our recommended best practices for improving performance on the disk subsystem: If your paging file is on the same partition as your operating system, use performance console to monitor the disk queue length of that particular array.<br />
<br />
766<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
If the queue length is more than two times the number of drives in that array, consider moving the paging file to a dedicated array. Always protect your data by using the appropriate RAID level. RAID-1E provides the best performance, while RAID-5 or RAID-5EE gives the best price and performance. Use the fastest drives possible. Use the largest drives possible. You get better performance if you use eight 146.8 GB drives than if you use eight 36 GB drives. By using the large drives, you use a shorter disk stroke than by using smaller capacity drives. However, do not sacrifice the number of drives. More drives offers greater performance improvements than disk stroke. Set your ServeRAID adapter stripe size to either 32 KB or 64 KB. The RAID stripe size should be either 32 KB or 64 KB for Domino servers on Windows 2000 or Windows Server 2003 systems. For servers where the operating system and the data share the RAID controller, it is recommended that you set that the stripe size before you install the server’s operating system. Because the optimal stripe size can only be calculated after the server has been built, it is difficult to know which stripe size is best for your server. Where you have both the operating system and the data on the same ServeRAID adapter, we recommend that you use a 32 KB stripe size. Important: When changing the stripe unit in a ServeRAID adapter, you are changing the stripe unit for the entire controller. You lose any data on the disk arrays that are attached to the adapter, so it is best that you do this before you install the operating system on the adapter. If you intend to change the stripe size of a ServeRAID card for an existing server, make sure that you have a recent full backup. Consider disabling ServeRAID cache. If the disk subsystem is loaded heavily, throughput is greater with the ServeRAID set to write-through mode rather than write back. For more information, see 9.6.8, “Disk cache write-back versus write-through” on page 214. If you are using transaction logging, ensure that the logs are on a dedicated RAID-1 array. For more information, see 23.4.8, “Enabling transaction logging” on page 811. Put only one logical drive on an array. For more information, see 9.6.5, “Logical drive configuration” on page 206.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
767<br />
<br />
23.1.5 Network subsystem With the use of Gigabit Ethernet controllers, the network subsystem is rarely a problem on Domino servers. You should bear in mind that if you are going to set up a Domino cluster, you should install a second network controller (if there is not already a second one). You can then use the first adapter for normal network access and the second adapter for cluster replication exclusively.<br />
<br />
23.2 Optimizing operating system performance The operating system that hosts Domino is responsible for allocating the system’s resources for Domino to use. Any time a Domino server needs more memory, CPU time, disk space, and so forth, it is up to the operating system to provide that memory. If those resources are not readily available in a timely manner, then Domino’s performance suffers as a result. Each operating system handles performance tuning a bit differently, but they all have some basic components in common. They all offer ways to analyze the utilization of CPU, memory, disk, and networking resources to determine if any improvements can be made.<br />
<br />
23.2.1 Operating system memory allocation A common misconception is that Domino can use all of the real memory on a system. So, if a system has 2 GB RAM installed, then Domino should be able to use up to that amount if necessary. However, this method is not how Domino uses real memory. To understand how much memory Domino can use in a given operating system requires an understanding of how operating systems allocate virtual memory. Each operating system implements virtual memory a bit differently, but conceptually, they are all very similar. Every operating system requires some memory dedicated for overhead (hardware control, paging operations, and so forth). They also need to make some memory available for applications. In order to keep the operating system safe from applications that can monopolize memory, every modern operating system divides virtual memory into two distinct areas:<br />
<br />
768<br />
<br />
Kernel mode<br />
<br />
Memory that is reserved for use by the operating system itself<br />
<br />
User mode<br />
<br />
Memory that is available to applications that are running on the operating system<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Note: When a 32-bit process such as Domino is started, it is allocated a 4 GB virtual memory space, regardless of the available virtual memory. The operating system then handles how that virtual address space is used. Some operating systems, such as Windows and Linux, create the kernel mode and user mode memory areas from that single 4 GB virtual memory space. Other operating systems, such as AIX, create a separate 4 GB virtual memory space for kernel mode memory and, therefore, allow processes to use almost all of the 4 GB memory space that was allocated when the process started. Within the user mode memory space, memory is divided further into shared and private memory: Shared memory<br />
<br />
Contiguous memory that can be referenced and utilized by more than one process<br />
<br />
Private memory<br />
<br />
A region of memory that is reserved solely for use by a single process<br />
<br />
Restriction: While a process can (and in Domino’s case, usually will) use both private and shared memory, the sum of the two cannot go beyond the user memory limit that is imposed by the operating system. Knowing that each process is limited to a 4 GB virtual memory space, Table 23-1 lists the default memory limits for various operating systems. Table 23-1 Default memory limits<br />
<br />
Operating system<br />
<br />
Default user address space limit<br />
<br />
Shared memory limit<br />
<br />
Private memory limit<br />
<br />
Windows (32-bit)<br />
<br />
2 GB<br />
<br />
2 GB<br />
<br />
2 GB<br />
<br />
Linux<br />
<br />
3 GB to 4 GB<br />
<br />
3 GB to 4 GB<br />
<br />
3 GB to 4 GB<br />
<br />
AIX<br />
<br />
3.75 GB<br />
<br />
2.25 GB<br />
<br />
512 MB<br />
<br />
Solaris<br />
<br />
3.9 GB<br />
<br />
3.9 GB<br />
<br />
3.9 GB<br />
<br />
OS/390®<br />
<br />
2 GB<br />
<br />
2 GB<br />
<br />
2 GB<br />
<br />
The User mode address space limit represents the absolute maximum amount of Virtual Memory that is available to a Domino Server. If a server has more RAM than can be used, then you can add more Domino Server partitions to make better use of the available memory.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
769<br />
<br />
Attention: By default, Windows reserves 2 GB of virtual address space for kernel-mode memory, and 2 GB to user-mode memory. You can change this setting using the /3GB switch in the boot.ini file. Using this setting tells the operating system to make 3 GB available for User memory and only 1 GB for Kernel memory. For an application to take advantage of this option, it needs to be large address aware. Without this awareness, a 32-bit application will only make use of 2 GB worth of its address space. Additionally, only 1 GB will have been allocated to the kernel, leaving 1 GB worth of address space completely unavailable for use at all. Beginning with release 7.0.1, Domino is large address aware in Windows and, therefore, can take advantage of more memory than previous Domino releases. For information about how to set this switch, see “Setting the /3GB switch” on page 835. As with all hardware subsystems, you should monitor memory allocation regularly using the Windows Performance console as described in 14.1, “Performance console” on page 472. If the average memory utilization exceeds 70%, then you run the risk of the server paging excessively, which can affect server performance adversely. You can find more information about operating system memory usage in Chapter 8, “Memory subsystem” on page 129.<br />
<br />
23.2.2 System cache As an administrator, you can influence how Windows Server 2003 allocates the physical memory that it has at its disposal. What you specify depends on the amount of memory in the system and how much of it is being used. The configuration of the system cache is dependent on the amount of physical RAM and the amount of memory that being used in any particular server. You have two possible configurations: Maximize the amount memory that is available to Domino by limiting the system cache to 512 MB Allow the system cache to grow to its maximum size of 960 MB If you have a system with more than 2 GB of memory installed, then you might want to select the second option. Domino will not use more than 2 GB of RAM. So, any spare memory might be best used by allocating it to the system cache. If memory is at a premium, then the first option is more desirable.<br />
<br />
770<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To configure the system cache, follow these steps: 1. Right-click My Network Places and click Properties. 2. Right-click any network connection and choose Properties. This setting affects all LAN interfaces, so which LAN connection you choose is not important. 3. Select File and Print Sharing for Microsoft Networks. 4. Click Properties. The window shown in Figure 23-2 opens. – Select Maximize data throughput for network applications if you want to reserve the maximum amount of memory for Domino. – Select Maximize data throughput for file sharing if you want the System Cache to be allowed to grow to 960 MB.<br />
<br />
Better for file servers and servers with more RAM<br />
<br />
Better for servers with small amounts of available RAM<br />
<br />
Figure 23-2 Configuring the Server Optimization<br />
<br />
5. Click OK and then Close. You can achieve this configuration by updating two registry values. However, we recommend that you use the GUI procedure that we have described instead.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
771<br />
<br />
Maximize data throughput for network applications: Set the following key to 0: HKLM\System\CurrentControlSet\Control\SessionManager\ MemoryManagement\LargeSystemCache Set the following key to 3: HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\Size Maximize data throughput for file sharing Set the following key to 1: HKLM\System\CurrentControlSet\Control\SessionManager\ MemoryManagement\LargeSystemCache Set the following key to 3: HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\Size<br />
<br />
23.2.3 Application responsiveness With Windows servers, it is possible to give foreground applications, those that are running at the console, higher priority than applications that are running as a service. Because Domino should be configured to run as a service, Windows should be optimized for background applications. Windows Server 2003 is optimized for background services by default. If you want to ensure that this setting is correct then follow this procedure: 1. Right-click the My Computer icon and select Properties. 2. At the System Properties dialog box select the Advanced tab. 3. In the Performance frame click Settings. 4. At the Performance Options dialog box, select the Advanced tab. You should now see a dialog box similar to the one shown in Figure 23-3 on page 773.<br />
<br />
772<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 23-3 Advance Performance Options settings<br />
<br />
5. In the Processor scheduling frame, ensure that the Background services radio button is selected. Click OK when you are done. 6. Click OK to close the System Properties dialog box.<br />
<br />
23.2.4 Domino on Linux There are a number of kernel parameters that can affect Domino server performance. For more information about tuning these parameters, refer to Chapter 12, “Linux” on page 371. fs.file-max This parameter specifies the maximum number of file handles that Domino can open. The sysctl fs.file-max command shows the current value. Ensure that the value is at least 49152 (48 x 1024). You might find that the default value is higher than this value. In that case, leave the value unchanged. sysctl -w fs.file-max=49152<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
773<br />
<br />
kernel.shmmni This parameter specifies the maximum number of shared memory segments for the operating system. Ensure that the value is at least 8192 by checking its value using sysctl kernel.shmmni. If the value is smaller, set it to 8192 with the following command: sysctl -w kernel.shmmni=8192 kernel.threads-max This parameter specifies the maximum number of threads for the operating system. Ensure that the value is at least 8192 by checking its value using sysctl kernel.threads-max. If the value is smaller, set it to 8192 with the following command: sysctl -w kernel.threads-max=8192 Set noatime on the file systems that include the Domino data directories, as described in “Accessing time updates” on page 398.<br />
<br />
23.3 Optimizing Lotus Domino performance Lotus Domino installed without modifications is optimized for most configurations. However, there are some tuning parameters in specialized situations or for specific server roles that can give your Lotus Domino server added performance and that can protect the server from overloading. This section discusses some of those optimization features. Tip: The Lotus Notes Domino server is controlled by values that are stored in the notes.ini file. We recommend that you take a backup of your NOTES.INI file before you make any modifications, in case your settings are not accepted by the system.<br />
<br />
23.3.1 Changing statements in the notes.ini file To tune various parameters of Lotus Domino server, you need to manipulate statements in the notes.ini configuration file. Before making changes to the NOTES.INI file, use the SHOW CONFIGURATION server command at the console to check the existing settings.<br />
<br />
774<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To edit the settings in notes.ini, use one of the following methods: Using a Server Configuration document The Server Configuration document is a form that exists in the Lotus Domino server Name and Address Book. Use a Server Configuration document to specify settings for a single server, a group of servers, or all servers in a domain. You cannot specify all notes.ini settings in a Server Configuration document. The Set/Modify Parameter list in the Server Configuration document lists default settings that you can specify. Tip: When you set variables using the Server Configuration document in the server’s Name and Address Book database, you will get additional information about the particular variable’s usage. Using the SET CONFIGURATION server command at the console The SET CONFIGURATION server command writes the new setting to the notes.ini file. If you previously specified the setting in any Server Configuration document that affects this server, Lotus Domino writes the new setting to the Server Configuration document specific to the server being configured or creates a new document if necessary. Editing the NOTES.INI file directly in a text editor We do not recommend editing the notes.ini file directly because of the possibility of introducing errors into the file and impairing the operation of the Lotus Domino server. In most cases, you must restart the Lotus Domino server to put your changes into effect.<br />
<br />
23.3.2 Configuring server tasks Each task increases the server's load and can affect server performance adversely. Minimizing the number of server tasks that run on the server, the frequency at which they run, and the time in which they run allows you to increase the performance of the server. The Lotus Domino server tasks that are launched automatically by the server are set in the notes.ini variables. For example: ServerTasks=Router,Replica,Update,Amgr,AdminP,CalConn,Event,Sched,Stats ,HTTP,DIIOP,IMAP,POP3,NNTP,DECS,maps ServerTasksAt1=Catalog,Design ServerTasksAt2=UpdAll,Object Collect mailobj.nsf ServerTasksAt3=Object Info -Full ServerTasksAt5=Statlog<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
775<br />
<br />
Each of these variables controls the schedules for automatic server and database maintenance tasks. The time is entered in a 24-hour format, where 0 is 12:00 a.m. and 23 is 11:00 p.m. In this example, Catalog and Design tasks would initiate at 1:00 a.m., and the Statlog task would initiate at 5:00 a.m. You can improve performance significantly by removing tasks that are not appropriate to the server if it is a specialized server. The following are some suggestions on how to increase performance related to Lotus Domino server tasks. Remove the Stats server task (installed by default) This task is installed only for backward compatibility with Notes Release 2 or later servers that are set up for statistics collecting and can be removed (although it takes up very few resources). This task is the server task that produces the following messages on the server console once each hour: Stats agent started Stats agent shutdown Remove the Replicator and Router tasks You can remove both of these tasks if they are not used on the server, because each of these tasks take up a fair amount of server resources when loaded. For example, if you have only one Lotus Domino server in your organization that is used for both applications and mail routing, you might not need the Replicator task, because you do not have any other servers to replicate from (and because clients will be replicating from the Lotus Domino server not vice versa). Another example might be that you have a hub replication server that never routes mail. In this case, you can remove the router task. Be careful with this example because, although the server might not be used for routing mail between users, you might have some mail-enabled Lotus Domino database applications, in which case you will need the Router task and should not remove it. Note: Do not remove the Update task from a server. If you do so, the Public Address Book will not update. Carefully choose the times when server tasks are run Daily server tasks should be run when other server tasks are not running and at times when few or no users are using the Lotus Domino server, such as before users begin using the server in the morning, during lunch time or after normal working hours. This allows the maximum amount of server resources to be available for each server task that is executing currently and for user<br />
<br />
776<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
sessions. Examples of such server tasks are Design, Catalog, Statlog, and customized Notes API programs that need to run only once a day.<br />
<br />
23.3.3 Optimizing database performance Database performance affects the overall Lotus Domino server performance in the most significant way because Lotus Domino uses databases for most of its activities. However, a complete description on database performance is simply to broad for this book. The Domino Designer® 7.0 Help database also provides useful information: http://doc.notes.net/domino_notes/7.0/help7_designer.nsf Additionally, you can find two excellent articles on tuning the performance of Notes/Domino 7 databases online at: http://www-128.ibm.com/developerworks/lotus/library/notes7-applicationperformance1 http://www-128.ibm.com/developerworks/lotus/library/notes7-applicationperformance2<br />
<br />
23.3.4 Defining the number of databases cached simultaneously If your server has sufficient memory, you can improve the performance of the server by increasing the number of databases that Lotus Domino can cache in memory at one time. To do so, use the NSF_DbCache_Maxentries statement in the notes.ini file. The default value is 25 or the NSF_Buffer_Pool_Size divided by 300 KB, whichever value is greater. The maximum number of databases that can be cached in memory is approximately 10 000. For short intervals, Lotus Domino stores up to 1.5 times the number that is entered for this variable. Monitor the Database.DbCache.Hits statistic on your server. This value indicates the number of times a database open request was satisfied by finding the database in cache. A high value indicates database cache is working effectively. If the ratio of Database.DbCache.Hits to InitialDbOpen is low, you might consider increasing NSF_DbCache_Maxentries. To set the number of databases that a server can hold in its database cache at one time, set the notes.ini value as follows: NSF_DbCache_Maxentries = [number]<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
777<br />
<br />
In special circumstances, you might also want to disable the database caching. The database cache is enabled by default. To disable database caching, enter the following syntax on the Domino console: Dbcache disable The database cache keeps databases open. Use this command to disable the database cache when you need exclusive access to a file that might be in the database cache. For example, to run an application such as a virus checker or backup software, disable the cache. To enable the cache, restart the server.<br />
<br />
23.3.5 Scheduling utilities To optimize your Lotus Domino performance, schedule CPU-intensive utilities during non-peak processing times. Domino administrative utilities — Updall, Compact, and Fixup — that run against a large number of databases (or even a small number of very large databases) can be very CPU intensive. In these situations, you should not run these utilities during peak processing periods because of the likelihood of elongated user response times. In general, the following schedules are recommended for these utilities: You should run updall nightly to update all view and full-text indexes for all Domino databases. You should run compact against all databases on a weekly basis. Useful options include the following, but a full list of options can be found by entering compact /? -B -b -S nn<br />
<br />
All unused or white space is recovered and file sizes are reduced. White space is recovered, but file sizes are not reduced. Where nn is a percentage value 1-99. This instructions the command to only compacts databases when white space equals or exceeds the specified percentage.<br />
<br />
The -b option is the recommended compaction method, but there are many others documented in the Domino Administration Help database. The style that you choose is dependent upon your goals of reducing disk space, reorganizing databases to improve performance, or other reasons. Keep in mind that running compact against a database with 5% to 10% of white space will not realize any noticeable performance gains. If running transaction logging in conjunction with a backup tool that provides point-in-time database recovery, then you need to consider the style, frequency, and schedules of compactions in your backup, recovery, and strategy planning. Run fixup only as needed against selected databases to fix corrupted views and documents. For databases that are transaction logged, you should not<br />
<br />
778<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
need to run fixup (unless you are requested to do so by the IBM Support Center) if the Translog_AutoFixup parameter in the NOTES.INI file is set to 1. This setting causes Domino to repair any databases with corruption issues automatically. If running transaction logging in conjunction with a backup tool that provides point-in-time database recovery, a full backup of any databases that have been repaired with fixup is required immediately after running this utility. Tip: Neither the compact nor updall utilities are destructive by nature, in that no actual data is removed during when you execute them. Thus, it is safe to run them on a regular basis. Unlike Compact and Updall, Fixup does have a destructive component to it. If Fixup is unable to repair a corrupt element that it finds, it simply deletes that element. Therefore, if you run Fixup on a schedule over the weekend, there is a chance (albeit a slim one) that the utility will delete documents from a database with no prior warning to administrators that the documents were corrupt. By running Fixup manually, when indications of corruption are already present, administrators should be better prepared to address the problem and not be surprised by it. Updall, Compact, and Fixup are intended to execute serially to avoid locking and other issues. Each of these utilities offers the flexibility to run against specific databases, all databases within a folder, and all databases on a Domino server. If CPU cycles are available during off-shift periods, then multiple instances of these utilities can be run against different folders to shorten processing windows as long as there is no overlap of folder processing between any of the utilities. You should schedule backups of a large number of databases and, in some cases, recoveries of individual databases to run off-shift and at different times than other Domino utilities because of the potentially large demand on system resources.<br />
<br />
23.3.6 Optimizing database index update One of the most common, preventable causes of poor response time and performance of Domino servers is excessive and unnecessary activity of the Domino Update task. This task updates and rebuilds the indexes of Domino database views and full text indexes. It runs in the background and improves response time and performance by ensuring that when a user opens a database<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
779<br />
<br />
view, the user does not have to wait for it to be indexed. Ironically, it often turns out to have the opposite effect. This section explains how the Update task is designed to work. It looks at how Domino administrators and designers can monitor and control it to improve server performance. It also explains the potential impact to performance when the Update task is running excessively. It provides a hint of the kind of improvements you might see by correcting the problem. The Update task, also known as the indexer, is responsible for automatic updates of both view and full text indexes. It is loaded at server startup, by default, and runs continuously, checking its work queue for views and folders that require updating. Update maintains two work queues: an immediate queue and a deferred queue. Deferred update requests are held for 15 minutes before they are processed so that requests to update the same database in that time are ignored as duplicate requests. Updater queues can grow if the server has a high update rate because of heavy database use. Important: Immediate updates do not necessarily occur immediately. They are simply added to the Pending queue where they are processed as soon as an indexer thread is available to process them. By default, the indexer waits five seconds between each database update operation that it performs. When a request is made to update a view, the view is only updated if there are at least 20 note changes since the last update and if the view has been accessed within the last seven days. Tip: Domino 7 allows a separate thread to be created solely for full text updates, if needed. This separate thread is responsible for updating only full text indexes, so view updates continue to occur even if a large full-text index is being rebuilt. To enable the full text indexer-only thread, add the following line to the notes.ini file: UPDATE_FULLTEXT_THREAD=1<br />
<br />
780<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Note: A separate directory indexer thread is spawned automatically by update to handle view updates of any local or remote Domino Directory or Extended Directory Catalog that a server uses for directory services. This directory indexer thread runs every minute and does not lock the views during updates, permitting new server sessions while this task is running. Important: If the views are being rebuilt (Updall -R) then the views are locked, preventing authentication until the rebuild is completed. For this reason, you should defer view rebuilds in the Domino Directory until after peak hours.<br />
<br />
Full-text indexes Full-text indexing options and update frequency can affect server disk space and processing speed. So, they deserve special consideration to ensure that they perform adequately without hurting overall server performance. You must periodically update full-text indexes on servers to keep them synchronized with changes to the databases. When you create an index, you can either accept the default schedule for updating it (nightly at 2 AM) or specify a different schedule. You can change this schedule at any time using database properties. Figure 23-4 shows the different update frequencies as well as the full text settings for a particular database.<br />
<br />
Figure 23-4 Update frequency options<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
781<br />
<br />
Table 23-2 and Table 23-3 explain the indexing options and update frequencies, respectively, available for full text indexes. Table 23-2 Indexing options<br />
<br />
Indexing option<br />
<br />
Description<br />
<br />
Index attached files<br />
<br />
Indexes attachments. Also choose either With found text to include just the ASCII text of attachments or With file filters to include the full binary content of attachments. Note: Choosing With found text creates the index faster than choosing With file filters but is less comprehensive.<br />
<br />
Index encrypted fields<br />
<br />
Indexes text in encrypted fields.<br />
<br />
Index sentence and paragraph breaks<br />
<br />
Includes sentence and paragraph breaks in addition to word breaks to allow users to do proximity searches.<br />
<br />
Enable case sensitive searches<br />
<br />
Allows searches by exact case match. Note: This option increases the size of the index by about 15%, as each word must be indexed twice—for example, apple and Apple.<br />
<br />
Table 23-3 Update frequencies<br />
<br />
Update frequency option<br />
<br />
Updates occur<br />
<br />
Daily<br />
<br />
Nightly when the Updall server program runs by default at 2 a.m.<br />
<br />
Hourly<br />
<br />
Every hour, as scheduled by the Chronos server task. Note: Can be disabled using the setting Debug_Disable_Chronos=1 in the notes.ini<br />
<br />
Immediate<br />
<br />
As soon as possible after you close the database.<br />
<br />
Scheduled<br />
<br />
As scheduled by a Program document for the Updall server task in the Domino Directory. Note: If you select the Scheduled option and do not create a Program document for Updall, scheduled updates do not occur.<br />
<br />
Monitoring Update performance If a server has a high update rate due to heavy application database use, a large number of mail users, or a large volume of mail, the default resource usage configuration can cause the updater queues to become large, typically indicating that views and full-text indexes are not up-to-date.<br />
<br />
782<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To determine whether the updater queues are large, examine the queue length statistics that are available in Lotus Domino 7 using the following command on the Domino server console: show stat update Table 23-4 lists the available statistics and their definitions. Table 23-4 Available statistics<br />
<br />
Statistic<br />
<br />
Definition<br />
<br />
Update.DeferredList<br />
<br />
Number of requests for view updating or full text indexing on the deferred queue<br />
<br />
Update.DeferredList.Duplicates<br />
<br />
Number of requests for view updating and or full text indexing avoided because they were already waiting on the deferred queue<br />
<br />
Update.DeferredList.Max<br />
<br />
Maximum number of requests waiting for view updating or full text indexing on the deferred queue<br />
<br />
Update.DeferredList.Processed.AllViews<br />
<br />
Number of all view updates processed from the deferred queue<br />
<br />
Update.DeferredList.Processed.Compactions<br />
<br />
Number of compactions processed from the deferred queue<br />
<br />
Update.DeferredList.Processed.SingleViews<br />
<br />
Number of single view updates processed from the deferred queue<br />
<br />
Update.FullTextList<br />
<br />
Number of requests on the full text index queue<br />
<br />
Update.FullTextList.Duplicates<br />
<br />
Number of requests for full text indexing avoided because they were waiting on the full text index queue<br />
<br />
Update.FullTextList.Max<br />
<br />
Maximum number of requests waiting for full text indexing<br />
<br />
Update.FullTextList.Processed<br />
<br />
Number of full text indexing requests processed<br />
<br />
Update.NAB.Updates<br />
<br />
Number of Domino Directory view updates processed<br />
<br />
Update.PendingList<br />
<br />
Number of requests for view updating or full text indexing on the immediate queue<br />
<br />
Update.PendingList.Max<br />
<br />
Maximum number of requests waiting for view updating or full text indexing on the immediate queue<br />
<br />
Update.PendingList.Processed.AllViews<br />
<br />
Number of all view updates processed from the immediate queue<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
783<br />
<br />
Statistic<br />
<br />
Definition<br />
<br />
Update.PendingList.Processed.Compactions<br />
<br />
Number of compactions processed from the immediate queue<br />
<br />
Update.PendingList.Processed.SingleView<br />
<br />
Number of single view updates processed from the immediate queue<br />
<br />
Attention: Because deferred updates are delayed 15 minutes by design, it is important to monitor the Update.Deferred* statistics over time, instead of acting on a single set of data. If either the Update.FullTextList or Update.PendingList statistics show more than just a few updates in the queue, then updates might be backing up in the queue. For example, the following statistic output indicates an indexer with no real backlog: Update.PendingList = 8 In contrast, the following statistic indicates that updates are definitely backing up in the queue: Update.PendingList = 2738 If this backup happens, it is possible to tune the indexer’s performance by using the following notes.ini settings: Update_Access_Frequency This setting, in days, allows updates to occur against views that have been accessed within the number of days specified here as opposed to the default is seven (7). Lowering this number reduces the frequency of updates to rarely accessed views, allowing faster updates to more frequently used views instead. Update_Note_Minimum This setting controls how many changes must exist for a view update to occur. The default is 20. Lowering this number forces more frequent updates of views to occur. Update_Suppression_Time Use this variable to specify the length of time that deferred update requests are held in queue before being serviced. The default is 15 minutes. Lowering this value causes deferred updates to occur more frequently.<br />
<br />
784<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Update_Suppression_Limit Use this variable to limit the size (number of requests) in the Update task's deferred requests queue. The default is 32767. Update_Idle_Time This setting, in seconds, changes the amount of time the indexer waits between database operations. The default is 5 seconds. Lowering this will allow the indexer to process updates more frequently, and as a result will consume more resources. If it is necessary to get more granular, you can use Update_Idle_Time_MS to make changes in milliseconds. Note: You can use FTUpdate_Idle_Time and FTUpdate_Idle_Time_MS to control separately the delay between operations that are performed on full text operations if the separate full text thread is being used (UPDATE_FULLTEXT_THREAD=1). Also, if the system’s resources allow, you can start additional Update tasks on the server, by adding an Updaters line in the notes.ini file. It is recommended that a maximum of one Update task be run per CPU. So, to enable four Update tasks on a Domino server, add the following to the notes.ini file: Updaters=4 In some situations, it might make sense to disable automatic view or full text updates completely. For example, if fast access to data is more important than the accuracy of the full-text index. In these cases, it is possible to disable automatic view and full text indexing if needed by adding settings to the server’s notes.ini file. UPDATE_DISABLE_VIEWS=1 This setting disables view updates on the server. With this setting in place, views are updated automatically when a view is accessed, which can significantly increase the time to open a database, or view. UPDATE_DISABLE_FULLTEXT=1 This setting disables full text updates on the server. With any of these settings in place, you can use a program document to run Updall in order to update the indexes for specific databases if needed.<br />
<br />
Updall Updall performs the same basic functions as the Update task, but it is run on an as-needed basis, as scheduled using program documents, or by the ServerTasksAtxx line in the server’s notes.ini file. Unlike update, Updall accepts arguments to control its behavior.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
785<br />
<br />
Tip: A complete list of available options and their behavior is available in the Domino Administrator Help 7 database in the document titled, Updall options. Updall also purges deletion stubs from databases and discards view indexes for views that have been unused for 45 days, by default. To change when Updall discards unused view indexes when no criteria has been defined by the database designer, add the following to the server’s notes.ini: Default_Index_Lifetime_Days=number of days The default value for this variable is 45 days unless a database designer has specified a different lifetime by clicking Design → View Properties → Advanced Tab → Discard index to reach the dialog box that is shown in Figure 23-5.<br />
<br />
This value overrides the value of Default_Index_Lifetime_Days as specified in the notes.ini<br />
<br />
Figure 23-5 Setting an override for the index lifetime<br />
<br />
Lowering the number of days to a lower value can reduce the number of view indexes on a server and, therefore, reduce the amount of time and effort that the Indexer server task must spend on updating view indexes. It is the responsibility of the Updall task to remove the view index if a discard option is specified. In other words, if a Discard Index option is selected for a view, the index is not actually discarded immediately. Rather, the index is removed the next time the Updall tasks runs. For example, a view has a Discard View option of After each use. If you exit from a view at 1:00 p.m., the view is not removed until Updall is run (usually done at 2:00 a.m.).<br />
<br />
786<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
23.3.7 Domino memory manager Domino provides its own memory management mechanism that works in conjunction with the operating system memory manager to allocate memory dynamically to Domino related tasks as needed. Using its own memory manager provides several benefits including: Tracking dynamic memory allocations for debugging purposes Allowing more granular control over how certain allocations are done (based on size or purpose) Allowing better cross-platform compatibility, by allowing memory requests by Domino processes to be made using a standard set of platform independent APIs Improving performance by retaining most its memory for later use Even though Domino has its own memory manager, it still relies on the operating system to allocate memory to it in the first place. When Domino has allocated memory from the operating system, it will not typically release this memory but will hold onto it for future use. Some limited types of allocations are immediately returned to the operating, but a majority of allocations are persistent. On a typical Domino server, the vast majority of memory that Domino manages is shared. Furthermore, the majority of that shared memory is usually part of the Unified Buffer Manager (UBM), also known as the NSF Buffer Pool. The UBM can use 70% to 80% of the total memory that is allocated to Domino. For this reason, it is critical to effectively manage the UBM. To determine whether the UBM needs to be resized, monitor the following statistic: Database.BufferPool.PerCentReadsInBuffer A high percentage here (>95%) indicates that the UBM is large enough, although it might be still be larger than necessary. This situation means that some portion of memory will go unused, so it might be a good idea to implement some sort of controls over the UBM directly. Domino 7 provides multiple notes.ini settings that you can use to configure the UBM. You can use the NSF_BUFFER_POOL_SIZE_MB parameter in the notes.ini file to set the size of the UBM directly. Using any of the following settings in the notes.ini file affects how RAM is calculated on the system and, therefore, affects the UBM size: PercentAvailSysResources ConstrainedSHM or ConstrainedSHMSizeMB MEM_AddressableMem or MEM_AddressableMemSizeMB (ND 7.0.2)<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
787<br />
<br />
The following is the sequence Domino 7 uses for determining the physical RAM installed on a server. 1. Queries system for amount of installed RAM. For Windows and Linux, this is the full amount of installed RAM returned in ND7. 2. Applies the value of PercentAvailSysResource in the notes.ini file to installed RAM size. 3. Applies User Space Limit (as of Domino 7.0.2) (if lower than #2) Applies MEM_AddressableMemSizeMB if specified Applies default size if MEM_AddressableMem is set 4. Applies Constrained Shared Memory Usage (if lower than #3) Applies ConstrainedSHMSizeMB if specified Applies ConstrainedSHM if ConstrainedSHMSizeMB not specified Applies 4 GB cap if neither are specified The physical memory never goes above 4 GB. The maximum UBM size is then based on 3/8th calculated RAM from above unless, NSF_BUFFER_POOL_SIZE_MB is defined. Here are some comparisons of these settings that can help determine which is best suited for a particular need. PercentAvailSysResources – – – – –<br />
<br />
Typically used with multiple Domino Partitions on a single box Affects the calculation of RAM (and indirectly the UBM) Can specify what percentage of RAM each partition sees as installed Does not take into account inherent address space limitations Does not place a limit on shared memory usage outside of the UBM<br />
<br />
ConstrainedSHM and ConstrainedSHMSizeMB – – – –<br />
<br />
Can be used with either partitioned servers, or stand-alone Affect the calculation of RAM (and indirectly the UBM) Take into account inherent address space limitations Enforce a limit on overall amount of shared memory usage not just the UBM. – Enforced directly by the Domino memory manager during allocation of shared memory<br />
<br />
788<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
MEM_AddressableMem and MEM_AddressableMemSizeMB (7.0.2) – – – –<br />
<br />
Can be used with either partitioned servers, or stand-alone Take into account inherent address space limitations Affect the calculation of RAM (and indirectly the UBM) Do not place a limit on overall shared memory usage (can’t control overall shared memory usage with this setting)<br />
<br />
NSF_BUFFER_POOL_SIZE_MB – – – –<br />
<br />
Can be used with either partitioned servers, or stand-alone Does NOT take address space limitations into account Does NOT affect calculation of RAM, but does set limit on UBM size Does NOT place a limit on overall shared memory usage<br />
<br />
Note: ConstrainedSHM can generally replace the use of PercentAvailSysResources, although it is recommended to limit ConstrainedSHMSizeMB to no higher than 2048 on Linux systems. Setting ConstrainedSHM=1 enforces a limit on shared memory that varies between 1.5 GB and 3 GB depending on the platform. Tip: In most cases, any changes to the UBM size through these settings should be in an effort to reduce the overall size of the UBM, not enlarge it.<br />
<br />
23.3.8 Displaying images after documents To display documents quickly for a database that includes images, follow these steps: 1. Open the database. 2. Select File → Database → Properties. 3. Select the Basics tab.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
789<br />
<br />
4. Select the Display images after loading option (Figure 23-6).<br />
<br />
To display images after document text, select this option<br />
<br />
Figure 23-6 Displaying images after document selection<br />
<br />
Lotus Notes users can then read the text while the images load. If you do not load images after text, Notes loads images in the order in which they appear in a document. If an image appears first, Notes loads it before displaying text. With large images or slow connections, loading images in order can slow the display of the document. Note: Users can also choose Load images: On request in the Advanced section of a location document to display images only when users click them.<br />
<br />
23.3.9 Disabling the unread marks maintenance By default, Lotus Domino server maintains unread marks for each database, which takes system resources and can impact database performance. For many databases, such as mail files and discussion databases, unread marks are indispensable. However, for many databases, this feature does not serve a useful purpose. Databases such as LOG.NSF, the Domino Directory, and the Help and Administration databases all belong to this category. To disable the unread marks, follow these steps: 1. Open the database. 2. Select File → Database → Properties to open the Database Properties dialog window. 3. Select the Advanced tab.<br />
<br />
790<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
4. Select the Don’t maintain unread marks option (Figure 23-7).<br />
<br />
Figure 23-7 Advanced tab settings for a single database<br />
<br />
23.3.10 Optimizing of document table bitmap By default, during view updates and rebuilds, Domino searches each table for documents that appear in the view being updated. To update views more efficiently, do the following: 1. Open the database. 2. Select File → Database → Properties. 3. Select the Advanced tab. 4. Select the Optimize document table map option, as shown in Figure 23-7. This property associates tables with the forms that are used by the documents that the tables include. Then, during a view update, Domino searches only the tables that are associated with the forms that are used by documents in the view being updated. This improves the performance of view updates significantly, especially updates of small views within large databases as, for example, the Connections view in the Domino Directory. This property only works for views that use Form= as part of the selection criteria. There is a slight performance cost to maintaining the table/form<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
791<br />
<br />
association; however, when updating small views in large databases, the benefits offset the cost. If you select or deselect Optimize document table map, you must compact the database so that the setting takes effect. You can also run the compact server task with the -F or -f option to enable or to disable this property and then compact.<br />
<br />
23.3.11 Do not maintain Accessed document property The Document Properties box displays the property Accessed which can show the date a document was last modified or read. The Maintain LastAccessed property selection controls whether the Accessed property is updated if the last document access was a read. Maintaining the Accessed property for reads causes disk I/O that would not otherwise occur. By default, the database property Maintain LastAccessed is not selected, meaning the Accessed (In this file) property is not updated when the last document access was a read, but only when the last access was a document modification. You should consider selecting Maintain LastAccessed property only in exceptional cases, for example if you use the document archiving tool, available in the Database Properties box, to delete documents based on days of inactivity. You can prevent maintaining the Accessed document property as follows: 1. Open the database. 2. Select File → Database → Properties. 3. Select the Advanced tab. 4. Deselect the Maintain LastAccessed property option if selected (see Figure 23-7 on page 791).<br />
<br />
23.3.12 Disabling specialized response hierarchy information By default, every document stores information that associates it with a parent document or a response document. Only two @functions use this stored information. Maintaining this information has a significant, negative effect on database performance.<br />
<br />
792<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To improve database performance, disable the response hierarchy information in databases that do not use @AllChildren and @AllDescendants functions: 1. Open the database. 2. Select File → Database → Properties. 3. Select the Advanced tab. 4. Select the Don't support specialized response hierarchy option, as shown in Figure 23-7 on page 791. If you select or deselect the Don't support specialized response hierarchy property, you must compact the database so that the setting takes effect. You can also run the compact -h option to enable or to disable this property and then compact.<br />
<br />
23.3.13 Preventing headline monitoring Users can set up headline monitoring to automatically monitor databases for information that interests them. Monitoring a database this way affects performance, especially if many users do this. To prevent users from monitoring a database, perform the following steps: 1. Open the database. 2. Select File → Database → Properties. 3. Select the Advanced tab. 4. Select the Don't allow headline monitoring option, as shown in Figure 23-7 on page 791. Administrators can also use the Security section of a Server document in the Domino Directory to control headline monitoring at the server level.<br />
<br />
23.3.14 Limiting the number of entries in the $UpdatedBy field Every document includes an $UpdatedBy field that stores the names of users or servers who have edited the document. If the document has been heavily edited, the number of entries in the field can be large and both consume disk space and degrade the database performance. If there is no need to track update information, you can either limit the field size or disable the function totally by setting the value to 0.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
793<br />
<br />
You can set the number of entries for these fields for each database by following these steps: 1. Open the database. 2. Select File → Database → Properties to display the database properties window. 3. Select the Advanced tab. 4. Write the limit into Limit entries in the $UpdatedBy field, as shown in Figure 23-7 on page 791.<br />
<br />
23.3.15 Limiting the number of entries in the $Revisions field The $Revisions field of a document stores the dates and times when changes were made to the document. This field is used to resolve, for example, replication conflicts. By default, the field stores up to 500 edit sessions. When the limit is reached, the oldest entry in the list is dropped. You should consider limiting the entries in this field when the following characteristics apply: The database includes many documents. The database replicates often or has no replicas. Database documents are not subject to much editing. The recommended value is ten entries. If you set the limit to less than 10, you might see more replication or save conflicts. You can prevent the overwriting of deleted data in your databases. You can set the value for each database by following these steps: 1. Open the database. 2. Select File → Database → Properties. 3. Select the Advanced tab. 4. Type the limit into the Limit entries in the $Revisions fields box, as shown in Figure 23-7 on page 791.<br />
<br />
794<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
23.4 Improving mail performance Lotus Domino offers a variety of tuning options to adapt its mail subsystem to your needs. You might, for example, want to disable mail routing completely on hub replication servers or to configure your server for the best performance as a mail routing hub.<br />
<br />
23.4.1 Setting maximum mail transfer threads The MailMaxThreads setting determines the maximum number of concurrent processes that the mail router can create to perform its mail transfers efficiently. The default setting is one thread per server port. Using more router threads allows mail to be routed more quickly. With more router threads more than one router thread looks in the mail.box database and distributes mail documents at the same time, instead of only one router routing the mail.box messages one-by-one, which is especially important on mail hub servers. However, additional threads can increase the demand for server processing time. (You specify the MailMaxThreads setting in the Server Configuration document in the Domino Directory, as shown in Figure 23-9 on page 797.) Although it is often desirable to route mail as quickly as possible, it is also possible to have too many router tasks running. If you are using multiple router tasks, you should monitor the performance and use of each of the tasks to be sure that none of them remains idle most of the time. Extra router tasks can take up significant system resources. We recommend that you also temporarily change the mail logging setting to show detailed thread information, so that you can gain a better understanding of task usage. You specify this setting in the Server Configuration document in the Domino Directory, as shown in Figure 23-8 on page 796.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
795<br />
<br />
Mail logging settings<br />
<br />
Figure 23-8 Configuration Settings window<br />
<br />
The available options are: Minimal<br />
<br />
Only logs errors and warnings<br />
<br />
Normal<br />
<br />
Minimal, plus transfer output and delivery output<br />
<br />
Informational<br />
<br />
Normal, plus information about transfer threads and processes, and more detailed information about the transfers and deliveries<br />
<br />
Verbose<br />
<br />
Information, plus more detailed information about transfer queues<br />
<br />
By setting this information, you see the usage of various threads and whether adding more is more effective for your server.<br />
<br />
Setting maximum mail threads for local delivery The MailMaxDeliveryThreads setting determines the maximum number of threads the router can create to perform local mail delivery. The default number is one. Increasing this value can improve message throughput for local deliveries. The ideal number usually falls in the range of three to 25, depending on the number of local deliveries on your Lotus Domino mail server.<br />
<br />
796<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Setting maximum mail concurrent threads for a destination If the MailMaxConcurrentXferThreads value is set to one, mail between two servers is transferred sequentially. Setting the value to one can, however, create a serious bottleneck, heavily impacting Lotus Notes mail users. For example, sending any large mail delays the transfer of all other mail that is routed to the same destination until the first mail transfer ends. We recommend that you monitor the mail traffic and balance the value between the network throughput and size structure of mail usually sent by users. See Figure 23-9 and Figure 23-10 on page 798.<br />
<br />
MailMaxThreads<br />
<br />
MailMaxConcurrentXferThreads<br />
<br />
Figure 23-9 Setting maximum mail transfer threads in Lotus Domino server<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
797<br />
<br />
MailMaxDeliveryThreads<br />
<br />
Figure 23-10 Setting maximum mail delivery threads in Lotus Domino server<br />
<br />
Disabling per-user message caching by the IMAP task This setting disables per-user message caching by the IMAP task. It can improve the capacity of a server by reducing memory consumption. However, response time for some user operations might be slower. If this setting is omitted, IMAP per-user message caching is enabled. To disable per-user message caching by the IMAP task, set the following value to one (1): NoMsgCache = [0 or 1]<br />
<br />
Controlling type-ahead addressing Type-ahead addressing displays names that match the letters a user types in the To, cc, and bcc fields in a mail message. The Lotus Domino server completes the remainder of the recipient's address for you automatically. You can change or retype the address as needed. To save time, Domino enables type-ahead addressing by default. To save bandwidth and to improve server performance, you can disable type-ahead addressing. If you disable type-ahead addressing on a mail server, users can still use type-ahead to find addresses in their Personal Address Book or mobile Directory Catalog.<br />
<br />
798<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To enable or disable type-ahead addressing, complete the following steps: 1. Make sure you already have a Configuration document for the server or servers to be configured. 2. From the Domino Administrator, click the Configuration tab and expand the Messaging section. 3. Click Configurations. 4. Select the Configuration document for the mail server or servers for which you want to restrict mail, and click Edit Configuration. 5. Under the Basics tab, complete the Type-ahead field as shown in Figure 23-11. Then, save the document.<br />
<br />
Disabling server type-ahead improves the server performance; users rely on the Personal Address Book or Directory Catalog.<br />
<br />
Figure 23-11 Disabling type-ahead on Lotus Domino server<br />
<br />
Creating multiple mail.box databases In Lotus Domino server, you can continue to use a single mail.box database or improve performance significantly by creating multiple mail.box databases. The mail router task uses a database called mail.box to store incoming and outgoing messages. Any process trying to write to the mail.box database needs exclusive access to the database. In other words, if another process is reading the mail.box database, the router write process must wait until the reading process relinquishes the database. As the mail load on the server increases,<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
799<br />
<br />
contention for access to the mail.box database creates a potential bottleneck that can limit performance. With multiple mail.box databases, Lotus Domino server can use multiple simultaneous processes in accessing mail.box data, improving mail routing performance. In the most scenarios four to 10 mail.box databases provide the best performance. Because disk contention is rarely an issue for the mail.box database, you do not need to put the multiple mail.box databases on different disks. However, you should spread user mail files across multiple disks to ensure that all mail files and mail.box databases are not on the same disk. Adding even one extra mail.box database can result in a noticeable performance improvement in mail routing. The benefit of increasing the mail.box database count from two is more marginal. To create multiple mail.box databases, complete the following steps: 1. Make sure you already have a Configuration Settings document for the server or servers to be configured. 2. From the Domino Administrator, click the Configuration tab and expand the Messaging section. 3. Click Configurations. 4. Select the Configuration Settings document for the mail server or servers for which you want to restrict mail, and click Edit Configuration. 5. Go to the Router/SMTP → Basics tab. 6. Under the Router/SMTP → Basics tab, complete the Number of mailboxes field as shown in Figure 23-12 on page 801. Then, save the document. 7. Restart the server.<br />
<br />
800<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Enter the number of mailboxes that you want to use on the server or servers.<br />
<br />
Figure 23-12 Setting up multiple mailboxes on the Lotus Domino server<br />
<br />
Note: You specify the number of mail.box databases in the Configuration Settings document. If you use the same Configuration document for multiple servers, Domino creates the number of mailboxes (mail.box databases) that you set in that document on each server using that document.<br />
<br />
23.4.2 Calendaring and scheduling resource use Lotus Domino provides a feature set to support calendaring and scheduling for Lotus Notes users. Calendaring and scheduling tasks can increase resource use to varying degrees in these basic ways: Workflow-oriented tasks, such as meetings, invitations, and responses create messages that are sent over the Domino messaging infrastructure. These messages tend to be short and are routed efficiently with minimal impact on the server. Free-time searches rely on the Notes remote procedure call (RPC) mechanism to get information from the clients’ home servers. Fundamentally, these searches are database queries to the busytime.nsf database, which are submitted through the client’s home server and possibly chained to other servers. The result set is small and is delivered with a minimum of network<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
801<br />
<br />
overhead. When requests need to be serviced by another server, the response is delivered efficiently without replication of data. The busytime.nsf is a very compact data structure that is purged regularly, thus minimizing data storage requirements. Typically, busytime.nsf can support hundreds of users with a few megabytes of data storage. The alarm system can add minimal overhead in conditions of extremely heavy usage. With alarms enabled, the alarm daemon process on the client polls the home server for new alarms every 10 minutes, opening the user’s mail database to do so. If all calendaring users on a server enable this option, a significant number of additional RPC requests are made against individual mail files. These requests are relatively insignificant compared to other mail activity on the server. However, administrators might notice additional transaction counts and log session activity because of this feature. Room and resource reservations are an area for consideration. Rooms and resources are implemented as separate Domino database applications. Resource requests are processed by an agent, which determines whether the room or resource is available. If it is available, the agent creates a reservation in the database and sends an acceptance notice back to the client. To minimize the latency between request and reservation, users can modify the Agent Manager setting in the notes.ini file, directing the Agent Manager to run more frequently than the default setting of every 30 minutes. If enough other, non-related user agents are enabled on servers experiencing heavy reservation system usage, the practice of decreasing the latency interval might affect overall server performance. Additional server processing bandwidth is required for specific calendaring processes: Calconn (Calendar Connector), which chains the free-time requests to other servers, and Sched (Schedule Manager), which updates the busytime.nsf database as users update their calendars. The free-time request is a chained RPC call to a remote server, so Calconn overhead is minimal. Typically, it is not even measurable by monitoring tools. In addition to basic requirements for processing tasks and serving data, you can also expect to use additional hard disk space. Calendaring and scheduling adds documents to users’ mail files. The exact amount of overheard depends on the usage profile. The impact of a small text message is minimal, for example. However, an organization that uses detailed meeting agendas with large object attachments might eventually start pushing disk storage limits. Of course, this problem pertains to all mail-enabled applications. Good systems management practices can help administrators monitor the growth of application files. Calendaring and scheduling has a minimal impact on network capacity and a small but measurable impact on server resources. In many cases, customers<br />
<br />
802<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
might find the total effect to be less than that of a traditional file-sharing system for calendaring. For a deeper analysis of C&S performance, test methodology and conclusions, see The Impact of Calendaring and Scheduling Workload on Domino Server Performance, which is available from: ftp://ftp.lotus.com/pub/lotusweb/product/domino/dominocands.pdf<br />
<br />
23.4.3 Minimizing logging activity Lotus Domino server includes a variety of powerful logging options that provide the ability to collect statistics on a wide range of activities. You can use these statistics for security reasons, performance tuning, or deployment tasks. However, the intensive logging activity can also have a significant negative impact on server performance. Thus, you should consider the following when implementing any logging activity: Limit the amount of information that is logged to log.nsf and the console log. For those parameters that provide various levels of logging (mail logging, log_replication, log_update, and so forth), choose the less verbose versions of logging, if possible, to reduce the amount of output and to potentially lower CPU requirements. Disable HTTP server logging to improve Web performance. Logging options are stored in the Server document. In the HTTP server<br />
<br />
Enable logging to section there are two fields: Log files and DOMLOG.NSF. Disabling both of these fields improves Web server performance. Disable parameters that start with debug when troubleshooting is complete. These parameters can add considerable CPU overhead and can generate a large amount of output. Do not log activity if you are never going to need that log information. By default, Statlog reports database activity to all database User Activity dialog boxes when it runs. Even if you disable User Activity reporting for a specific database, the next time Statlog runs, it enables recording in the dialog box again. To prevent Statlog from recording activity in User Activity dialog boxes automatically, add No_Force_Activity_Logging=1 to the notes.ini file. Then, you can enable activity recording per database, as needed.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
803<br />
<br />
Because recording activity in the User Activity dialog box adds 64 KB to the size of each database, disabling automatic activity recording saves disk space on the server. The syntax of this variable is as follows: No_Force_Activity_Logging = [0 or 1] Note: If you use No_Force_Activity_Logging, Statlog still reports activity to the log file. Even if the server administrator uses the No_Force_Activity_Logging setting in the notes.ini file to disable automatic activity recording in databases, you can enable recording for a single database and use this logging option more selectively, as shown in Figure 23-13. To enable recording for a single database: a. b. c. d.<br />
<br />
Open the database and choose File → Database → Properties. Select the i tab, and then click User Detail. Select Record Activity to enable activity recording. Click OK.<br />
<br />
Enable or disable activity recording per database Figure 23-13 Selective enabling of database activity recording<br />
<br />
23.4.4 Improving Agent Manager performance The Agent Manager controls when agents run on a server. Every time an agent runs, it uses server resources. To control when scheduled and event-triggered agents run, you specify settings in the Server document and in the notes.ini file. Customizing when agents run can conserve server resources, but it can also delay when agents run.<br />
<br />
804<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Controlling when or how often the Agent Manager runs agents In general, the more frequently agents run, the sooner they perform their tasks. Running agents more frequently, however, can increase demand on server resources and can adversely affect overall system performance. When you create or modify an event-triggered agent, the Agent Manager schedules it to run immediately, which ensures that the agent can quickly process new documents. These notes.ini settings let you specify a time interval between subsequent runnings of the agent. The time interval can prevent repeated runnings of the agent, for example, because of a rapid series of triggering events. Table 23-5 describes settings to control when and how often the Agent Manager executes agents. Table 23-5 Controlling when and how often the Agent Manager runs agents<br />
<br />
Statement = [minutes]<br />
<br />
Description<br />
<br />
AMgr_DocUpdateEventDelay default = 5 minutes<br />
<br />
Delay of execution that the Agent Manager includes in the schedule for a document update-triggered agent after a document update event. A longer interval results in the agent running less often, thus reducing demand for server time.<br />
<br />
AMgr_NewMailEventDelay default = 1 minute<br />
<br />
Delay of execution that the Agent Manager includes in the schedule for a new mail-triggered agent after a new mail message is delivered. A longer interval results in the agent running less often, thus reducing demand for server time.<br />
<br />
AMgr_DocUpdateAgentMinInterval default = 30 minutes<br />
<br />
The minimum elapsed time between executions of the same document update-triggered agent. Entering an interval can result in the agent running less frequently, reducing server demand.<br />
<br />
AMgr_NewMailAgentMinInterval default = 0 (no interval between executions)<br />
<br />
The minimum elapsed time between executions of the same new mail-triggered agent. Entering an interval can result in the agent running less frequently, reducing server demand.<br />
<br />
Note: Setting the parameters in Table 23-5 and other Agent Manager variables to zero does not completely eliminate the delay. A built-in delay always exist.<br />
<br />
Controlling how quickly the Agent Manager queues agents The Agent Manager checks periodically to see whether it has any new agents that it needs to schedule. These notes.ini settings control how quickly an agent<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
805<br />
<br />
gets into the schedule queue. Table 23-6 shows which settings control how quickly the Agent Manager queues agents. Table 23-6 Controlling how quickly the Agent Manager queues agents<br />
<br />
Statement = [minutes]<br />
<br />
Description<br />
<br />
AMgr_SchedulingInterval default = 1 minute<br />
<br />
Delay between runnings of the Agent Manager's scheduler.<br />
<br />
AMgr_UntriggeredMailInterval default = 60 minutes<br />
<br />
Delay between runnings of the Agent Manager's check for untriggered mail.<br />
<br />
Controlling how many concurrent agents are running You can relieve a heavily loaded Agent Manager by allowing agents to run concurrently. To do this, modify the Max concurrent agents field in the Server Tasks/Agent Manager section of the Server document, as shown in Figure 23-14. Values greater than 1 allow more than one agent to run at the same time. Valid values are 1 through 10. Default values are 1 for daytime and 2 for nighttime.<br />
<br />
Maximum number of agents running concurrently Figure 23-14 Allowing agents to run concurrently on Lotus Domino<br />
<br />
806<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
23.4.5 Managing server sessions You might want to set the maximum number of sessions that can run concurrently on your server. This is configured to prevent the server overloading because of a large number of users connecting to your server simultaneously. You can also define an inactivity period after which the server terminates user connections automatically.<br />
<br />
Maximum sessions When a new user attempts to log on, if the current number of sessions is greater than the value of Server_MaxSessions (in the notes.ini), the Lotus Domino server closes the least recently used session. In order for a session to be considered for closing, it must have been inactive for at least one minute.For example, if this parameter is set to 100, and the 101st person tries to access the Lotus Domino server, the Lotus Domino server drops the least-used session from the server in favor of this new session. Reducing maximum server sessions: Reducing the Server_MaxSessions parameter to a specific number does not prevent the server from allowing more than that number of concurrent active users on the server but does drop the sessions soon after they become inactive. This action frees resources. Conversely, Domino does not close any session that has been idle for less than one minute regardless of the demand on the server. You can determine session details by using the following console command: SHOW STAT NET Example 23-1 shows sample output of this command. Example 23-1 Sample output from SHOW STAT NET<br />
<br />
> show stat net NET.TCPIP.BytesReceived = 13,512 NET.TCPIP.BytesSent = 663,232 NET.TCPIP.Sessions.Established.Incoming = 3 NET.TCPIP.Sessions.Established.Outgoing = 0 NET.TCPIP.Sessions.Limit = 65535 NET.TCPIP.Sessions.LimitMax = 65535 NET.TCPIP.Sessions.LimitMin = 10 NET.TCPIP.Sessions.Peak = 6 NET.TCPIP.Sessions.Recycled = 0 NET.TCPIP.Sessions.Recycling = 0<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
807<br />
<br />
Note: The server in this example had a single network port named TCPIP. If your server has a different port name, then this is shown after the word NET. The key session details you want to investigate are: Sessions.Established.Incoming Sessions.Established.Outgoing Sessions.Peak Note that these session values are per network port. So, if you have more than one network card in your server, you need to add up all session values. If your server is overloaded, you can use these values to determine the value of the Server_MaxSessions value. The format of the Server_MaxSessions parameter in the notes.ini file is: Server_MaxSessions = [number]<br />
<br />
Session timeouts You can specify the number of minutes of inactivity after which the server automatically terminates network and mobile connections. The default session timeout is four hours, and the minimum recommended setting is 30 to 45 minutes. A lower setting might affect server performance negatively. The ideal setting depends on factors such as server load and the number or concurrent users on the server. Inactive database server sessions do not take up many resources on a Lotus Domino server, but it is a good idea to minimize them anyway in order to regain the resources that they do take up. To do this, use the following notes.ini variable: Server_Session_Timeout = [minutes] Note: For mobile connections, XPC has its own internal timeout. If the XPC timeout value is shorter than the Server_Session_Timeout value, the XPC timeout takes precedence.<br />
<br />
23.4.6 Controlling user access To manage a Lotus Domino server efficiently, you might need to control the server access of your users. You can, for example, limit the maximum number of users who access your server to protect the server against overloading or define Lotus Notes user access values that override the client settings.<br />
<br />
808<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Limiting the maximum number of users To protect your server from overloading, you can set the maximum number of users who are allowed to access a server. When you set this value and the maximum user limit is exceeded, Notes users are prompted with a message at their workstation when they try to access a database on the server, as shown in Figure 23-15.<br />
<br />
Figure 23-15 Window displayed when maximum users number is exceeded<br />
<br />
To set the maximum number of users, set the following notes.ini value: Server_MaxUsers = [number] The default value is 0, which indicates unlimited access to the server by users.<br />
<br />
Controlling minimum mail poll time You can control the minimum allowable time in which Lotus Notes clients can poll for new mail. It is possible that your Lotus Domino server resources are being overtaxed by being constantly bombarded by requests for new mail from the Notes client machines if users have changed their default new mail notification check time from 15 minutes to a smaller number, such as 2 or 5. You can control the minimum frequency of these requests from the server by using the MinNewMailPoll notes.ini variable. This variable determines the minimum allowable checking time from clients regardless of the value specified on the client machines. No default is set during server setup. The syntax of this variable is as follows: MinNewMailPoll = [minutes]<br />
<br />
23.4.7 Improving replication performance Streaming replication is a new Domino 6 feature enabled by default on servers and clients. To use streaming replication, both parties involved in the replication must be running Notes Domino 6. Streaming replication is pull only. This feature uses a single streamed RPC to read a number of documents and attachments. This is an improvement over the non-streaming method of requesting and acknowledging one database note at a time.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
809<br />
<br />
Streaming replication saves time and consumes less CPU and network activity than replicating the same number of documents. Network latency is reduced, especially on slower links. Documents are also replicated in ascending size order (smallest first). This allows the client to abort a long replication, but still receive some documents (for example, mail). Partial replication is also supported, allowing an aborted replication to be re-initiated and to continue where it left off. Views and folders are also updated incrementally while a replication is in progress (as opposed to the end of replication). This allows users to begin responding to initial documents (for example, mail messages) while larger documents continue downloading. Here are some best practices to consider when managing system replication: Be careful about over scheduling the server. When the server becomes overloaded, calls back up, mail is not delivered, and users might receive poor service from the server. Check the log.nsf file to ensure that the databases are replicating properly, that mail is routing promptly, and that the server is not overloaded. If necessary, modify the connection documents and make adjustments until the problem is solved. Use selective replication to replicate only documents from a certain person, about a certain topic, or subsets of databases. If replications are taking too long, change the schedule to replicate more often so that there are fewer updates per replication. Schedule replication at off-peak hours. If you are replicating internationally, consider the time zones for the source and destination servers and plan accordingly. For the most dramatic improvement in performance, set up multiple replications so that a server, such as the hub, can replicate with multiple servers simultaneously. This should improve replication performance immediately. Use replication options to shorten replication times. For example, if a hub server replicates to 50 servers and you want to make sure an entire replication cycle occurs twice a day, limit the time the hub connects to each server. Be sure to check the log to see which databases replicate completely and which do not. Set the replication priority to high, medium, or low to replicate databases of different priorities at different times. Set up replication groups based on replication priority. Develop a policy that controls how database replicas get placed on servers. Creating unnecessary replicas consumes system resources.<br />
<br />
810<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Check the Statistics & Events database for events indicating server problems.<br />
<br />
Setting up multiple replication tasks You can improve server replication performance by running multiple replicator tasks simultaneously. By default, only one replicator task is executed. With a single replicator task running, if you want to replicate with multiple servers at the same time, the first replication must complete before the next one can start. Set the number of replicators by adding the following entry in the notes.ini file: Replicators = [number] In this entry, the value is the number of replicators. All other factors aside, we recommend that you set the number of replicator tasks equal to the number of spoke servers with which the hub replicates. However, you should not exceed 20 replicators, to avoid putting too much load on the server. If the server you intended to replicate is not a hub server, the recommended number of replicators should equal the number of processors on the server. Bear in mind that your network design can affect the optimal number of replicators for your implementation. If you are replicating over a single 256 Kbps frame relay connection to your spoke servers, for example, adding extra replicator tasks would not provide any performance improvement. This is because the bottleneck is not the replicator task but the network link itself. With a 155 Mbps ATM connection, you might obtain better utilization by changing the value.<br />
<br />
23.4.8 Enabling transaction logging With transaction logging, each transaction is transferred to its database not directly, but by posting sufficient information to complete the transaction to a high-speed access sequential log file. The transaction is finally posted to the database at some later time. You can recover data in the log that is not yet posted to the database in the event of a server failure. Tests have shown that enabling transactional logging gives a performance advantage of up to 20%, depending on the exact server configuration. The Lotus Domino transaction log is actually a set of sequential accessed files used to store the changes made to the databases. Sequential disk access to a dedicated physical disk is noticeably faster than random disk access.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
811<br />
<br />
Transaction logging disk optimization: For optimal performance, you should place transaction logs on a separate physical disk device. If you place the logs on the same device as the databases, you lose the benefit of sequential access, and there is no, or very little, performance improvement. Transaction logging also has system-level benefits. At the operating system level, all the file and disk operations are carried out by the kernel running in privilege mode. This requires a context switch to the kernel thread to handle the task of moving data to and from disk. Transaction logging improves the overall I/O throughput of Lotus Domino; therefore, it reduces kernel time and increases user time, which reduces overall processor time to do the same work. To enable or disable transaction logging for all Lotus Domino databases on the server, set the Transactional logging field to Enabled in the Transactional Logging tab of the Server document, as shown in Figure 23-16.<br />
<br />
Transaction Logging enabled<br />
<br />
Runtime/Restart performance<br />
<br />
Figure 23-16 Basic Settings window<br />
<br />
You might also want to tune the Runtime/Restart performance value based on your needs. Runtime/Restart performance specifies the trade-off between transactional log runtime and restart recovery time. It is set through the Runtime/Restart performance field in the Transactional Logging tab of the Server document, as shown in Figure 23-16.<br />
<br />
812<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
The available options are: Favor runtime. The system stores more database changes in memory and writes fewer changes to the transaction log. Having fewer writes to disk improves server runtime. Standard (default). Favor restart recovery time. The system stores fewer database changes in memory and writes more changes to the transaction log. Having more writes to the transaction log improves restart recovery time.<br />
<br />
23.4.9 Improving Web server performance The use of Domino as a Web hosting solution on System x servers has been enhanced, with developers rewriting and optimizing the Domino HTTP stack for improved performance. You can further enhance server performance and response times by modifying certain HTTP parameters. In order to optimize performance of the Lotus Domino Web server, it is ideal to gather a baseline for performance. You should first check and record the current server performance and response times. Then, consider applying the tuning tips described in this section.<br />
<br />
Internet Site documents Domino Internet Site documents are used to configure the Internet protocols supported by Domino servers. A separate Internet Site document is created for each protocol, Web (HTTP), IMAP, POP3, SMTP Inbound, LDAP, and IIOP, which is then used to provide protocol configuration information for a single server, or for multiple servers in a Domino organization. Specifically, you can create: Web Site documents. You create a Web Site document for each Web site hosted on the Domino server. LDAP Site documents. You create an LDAP Site document for LDAP to an organization in a directory. IMAP, POP3, and SMTP Site documents. You create an individual Internet Site document for each mail protocol for which you enter an IP address. IIOP Site documents. You create an IIOP Site document to enable the Domino IIOP (DIIOP) task on the server. This task allows Domino and the browser client to use the Domino Object Request Broker (ORB) server program.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
813<br />
<br />
Internet Site documents make it easier for administrators to configure and manage Internet protocols in their organizations. For example, prior to Domino 6, if you wanted to set up a Web site in your organization, it was necessary to configure each Domino server in the domain with Mapping documents, Web realms, and File Protection documents. If you had virtual servers and virtual hosts, you had to do the same thing for them. In Domino 6, you can configure a Web Site document so that all servers and hosts use it to get configuration information for a Web site, including mapping information, file protection information, and Web realm authentication information. The Domino server is configured to use Internet Site documents if Load Internet configurations from the Server or Internet Sites Documents option is enabled on the Basics tab of the server document. If the option is not enabled, the server defaults to the Server document settings to obtain configuration information for Internet protocols. We recommend that you enable this new function for ease of management. Internet Site documents are created in the Internet Sites view, which is used to help manage Internet protocol configuration information by listing the configured Internet Site documents for each organization in the domain.<br />
<br />
Specifying network timeouts on the Web server Open, inactive sessions can prevent users from accessing the server. Specify time limits for activities between the Domino Web server and clients or CGI programs so that connections do not remain open if there is no network activity between them. Network timeout options appear under the HTTP sub-tab of the Internet Protocols tab in the Timeouts section of the Server document, as shown in Figure 23-17.<br />
<br />
814<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Web server network timeout options Figure 23-17 Network timeout tuning for Lotus Domino Web server<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
815<br />
<br />
Table 23-7 describes the fields and values that you can use to tune network timeouts for the Lotus Domino Web server. Table 23-7 Setting the network timeout options for the Lotus Domino Web server<br />
<br />
Field<br />
<br />
Action<br />
<br />
HTTP persistent connections<br />
<br />
Specify whether you want to enable persistent HTTP connections on the Web server. These connections remain active under the following conditions: HTTP protocol is 1.1. The server application returns an HTTP response code less than 400. (If the server application returns an HTTP response code greater than or equal to 400, the connection will be closed by the server.) The HTTP request came through a proxy server. The client did not send a connection close header. The number of connections that the server can support is running low, or the number of connections queued for the thread processing the request is too large. If the connection is kept open, the following settings apply: The connection will be closed if the maximum number of requests per connection is exceeded. The connection is closed if the persistent timeout is exceeded. The connection is closed if no data is received by the server within the specified input timeout. The connection is closed if a complete request is not received within the specified request timeout. Note: Persistent connections require more server overhead than connections that are limited by network activity.<br />
<br />
Maximum requests per persistent connection<br />
<br />
Specify the maximum number of HTTP requests that can be handled on one persistent connection. The default is 5.<br />
<br />
Persistent connection timeout<br />
<br />
Specify the length of time for which you want persistent connections to remain active. The default is 180 seconds.<br />
<br />
Request timeout<br />
<br />
Specify the amount of time for the server to wait to receive an entire request. The default is 60 seconds. If the server does not receive the entire request in the specified time interval, the server terminates the connection.<br />
<br />
Input timeout<br />
<br />
Enter the time, in seconds, that a client has to send a request after connecting to the server. The default is 15 seconds. If no request is sent in the specified time interval, the server terminates the connection. If only a partial request is sent, the input timer is reset to the specified time limit in anticipation of the rest of the data arriving.<br />
<br />
816<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Field<br />
<br />
Action<br />
<br />
Output timeout<br />
<br />
Enter the maximum time, in seconds, that the server has to send output to a client. The default is 180 seconds.<br />
<br />
CGI timeout<br />
<br />
The maximum time, in seconds, that a CGI program started by the server has to finish. The default is 180 seconds.<br />
<br />
Specifying the number of threads used by the Web server An HTTP request is processed by a thread. A thread, in turn, can handle a number of network connections. You can specify the number of threads the Web server can process. In general, the number of threads specified is an indication of the number of users who can access the server simultaneously. If the number of active threads is reached, the Domino server queues new requests until another request finishes and threads become available. The more power your machine has, the higher the number of threads you should specify. If your machine spends too much time on overhead tasks, such as swapping memory, specify a lower number of threads. Thread options appear under the HTTP sub-tab in the Internet Protocols tab in the Basics section of the Server document, as shown in Figure 23-18.<br />
<br />
Web server threads options<br />
<br />
Figure 23-18 Threads tuning for the Lotus Domino 7 Web server<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
817<br />
<br />
Table 23-8 describes fields and values that you can use to tune threads options for the Lotus Domino Web server. Increasing HTTP active thread improves performance of Web-related workload such as iNotes™. However, you should proceed with caution when using a number larger than 100. Table 23-8 Setting threads options for Lotus Domino Web server<br />
<br />
Field<br />
<br />
Description<br />
<br />
Number active threads<br />
<br />
The number of threads you want active on the server at the same time. The default is 40.<br />
<br />
Restricting the amount of data users can send The HTTP POST and PUT methods enable users to send data to the Domino server. The Server record field Maximum size of request content is new for Domino 6 and sets a limit on the amount of data that can be sent using either POST or PUT. This limit is enforced for all POST and PUT methods, whether the target is a database, CGI program, or Java servlet, and applies to all Web sites. The Web Site document includes two additional settings that control POST and PUT methods that target a database (for example, filling in a form or uploading a file attachment). Formerly available in the Server document, for Domino 6, these settings been moved to the Web Site document so that you can specify different values for each Web site. To restrict the amount of data that can be sent to a Domino database: 1. From the Domino Administrator, go to the Configuration tab, expand the Web section, and click Internet Sites. 2. Choose the Web Site document that you want to edit, and click Edit Document. 3. Go to the Domino Web Engine tab. Under POST Data, complete the field shown in Table 23-9.<br />
<br />
818<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Table 23-9 describes fields and values that you can use to manage the size of user’s uploads to the Lotus Domino Web server. Table 23-9 Setting POST data options for the Lotus Domino Web server<br />
<br />
Field<br />
<br />
Action<br />
<br />
Maximum POST data<br />
<br />
Enter the amount of data in KB that a user is allowed to send to the Web site in a POST request that targets a database. The default is 0, which does not restrict the amount of data that users can send (however, the amount is still limited by the Server record setting “Maximum request content”). This limit applies to both the PUT and the POST HTTP methods. If users try to send more than the maximum allowed data, Domino returns an error message to the browser.<br />
<br />
File compression on upload<br />
<br />
Choose one: Enabled: To compress files before adding them to a database. Compressing files saves disk space on the server. Disabled (default): If clients use a browser that supports byte-range serving. You cannot download compressed files using Domino byte-range serving.<br />
<br />
Improving file download performance for Web clients Web clients can download a file that is attached to a page or that is in a server directory that is mapped by a URL. If a client is using a product that supports byte-range serving (available in HTTP 1.1 and later), the client downloads the file in sections (ranges of bytes) and tracks the progress of each file download. If an interruption occurs, the client can resume the download from the point where it was interrupted. Without byte-range serving, users must repeat interrupted downloads from the beginning. Domino is compatible with clients that support the HTTP 1.1 specification. The clients can be implemented in a variety of ways, for example, as browser plug-ins, applets, or stand-alone programs. Domino automatically uses byte-range serving if the Web client uses a product that supports this feature. No configuration is necessary.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
819<br />
<br />
However, attached files must be decompressed in order for clients that support byte-range serving to access them. When you attach a file, you must clear the Compress option. To verify that an existing attachment is decompressed, choose File → Document Properties, select the $FILE item, and verify that the Compression Type property is NONE, as shown in Figure 23-19. Must be NONE for byte-range serving.<br />
<br />
Figure 23-19 Checking if the attachment is compressed<br />
<br />
Managing Web agents You can specify whether Web application agents, that is, agents triggered by browser clients, can run at the same time. These include application agents invoked by the WebQueryOpen and WebQuerySave form events and for agents invoked by the URL command Penitent. If you enable this option, the agents run asynchronously. Otherwise, the server runs one agent at a time. You should set an execution time limit for Web application agents. The purpose of the time limit is to prevent Web agents from running indefinitely and using server resources. However, do not rely on this mechanism for the routine shutdown of agents. When the server shuts down an offending agent, resources that the agent was using (such as disk files) might be left open.<br />
<br />
820<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To run Web application agents: 1. Open the Server document you want to edit. 2. Choose Internet Protocols → Domino Web Engine. Under Web Agents, complete the fields as shown in Table 23-10. Table 23-10 Web agents settings<br />
<br />
Field<br />
<br />
Enter<br />
<br />
Run Web agents concurrently?<br />
<br />
Choose one: Enabled: To allow more than one agent to run on the Web server at the same time (asynchronously). Disabled (default): To run only one agent at a time (serially).<br />
<br />
Web agent timeout<br />
<br />
The maximum number of seconds (elapsed clock time) for which a Web application agent is allowed to run. If you enter 0 for the value (default value), Web application agents can run indefinitely. Note: This setting has no effect on scheduled agents or other types of server or workstation agents.<br />
<br />
Domino Web Access (iNotes) performance Domino 6 Web Access (iNotes) performance was a top priority for the Domino development team. The focus was placed on optimizing, streamlining, and removing bottlenecks in Domino Web Access and in the underlying Domino 6 code. The significant improvements made include: Rewriting and optimizing the Domino 6 HTTP stack for performance (this benefits any Web-based application, not just Domino Web Access). Introducing new Notes memory management techniques to make memory allocation faster and to improve memory utilization. Enhancing NSF database core code for performance. Optimizing Domino Web Access code, especially for Calendar and Scheduling (C&S). The R6iNotes workload implements C&S. Minimizing or eliminating semaphore contention in Domino code. Optimizing the formula/compute engine code. Results have shown that for Domino 6 on Windows 2000, the number of Domino Web Access active users increased by 40% with a CPU cost per user saving of 30%. For further information about Domino Web Access (iNotes) performance features, refer to the following article: http://www-128.ibm.com/developerworks/lotus/library/ls-D6PerfFeatures/<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
821<br />
<br />
Compress HTTP response data By default, Domino Web Access uses compression (GZIP format) to reduce network bandwidth consumption and provide better performance, particularly for users with slow network connections. This option is set using the Compress HTTP response data setting on the Domino Web Access tab of the Configuration Settings document (Figure 23-20).<br />
<br />
Compress HTTP response data Figure 23-20 Compress HTTP response data setting<br />
<br />
23.4.10 Network performance (compression) Network compression is an important performance feature offered in Lotus Notes/Domino 6. When you enable network compression, data is automatically compressed before it is sent over the network. This improves network performance, especially over slower line speeds. Notes/Domino network compression offers a number of immediate benefits. For example, by reducing the amount of data being transmitted, you can improve the performance of your routing and replicating hub servers, especially if they are<br />
<br />
822<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
currently laboring under heavy workloads. In addition, you can enable it by default, so all your users can take advantage of this functionality without having to select it themselves. Because network compression is a standard, out-of-the-box feature, it does not require any additional code, which helps simplify administration and requires fewer CPU resources to run. Of course, as with any new functionality, network compression raises some immediate questions: Just how much network bandwidth do you save with compression? How does network compression affect server performance? Do you need this feature if you are already using attachment compression? How will this work with encryption? Can encrypted data be compressed and decompressed? Note the following statistics about the benefits of network compression: A 35% to 52% reduction in data transferred from the server to client A 26% reduction in data transferred from server to server Network compression is performed by the Notes Remote Procedure Call (NRPC). NRPC compresses the data before it is transmitted over the network, using the LZ1 algorithm developed by Abraham Lempel and Jacob Ziv. For detailed information about the LZ1 algorithm, see the IBM article, A fast hardware data compression algorithm and some algorithmic extensions, which is available at: http://researchweb.watson.ibm.com/journal/rd/426/craft.html With network compression enabled on both the sending and receiving computers, the Notes client, Domino server, or both, will attempt to compress any data (for example, e-mail or replicated documents) before transmitting it over the network. Network compression works with mail routing, replication, or any other data sent through NRPC. This includes design elements in new or updated replica databases. Network compression works for both Domino-to-Domino and Domino-to-Notes sessions. Both the sender and receiver must be running Notes/Domino 6 and have network compression enabled. Otherwise, the data will be sent uncompressed. Data that is first encrypted cannot then be compressed. However, data that is first compressed can then be encrypted. With Notes Domino 6, this is not an issue. If you use port-level encryption in Notes Domino 6 combined with network compression, network data is automatically compressed first and then encrypted.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
823<br />
<br />
When you do this, you receive the full benefit of both compression and encryption. Network compression cannot further compress an already compressed attachment. This includes files compressed by Notes when attached (Huffman or LZ1 attachment compression) or in a compressed file format (such as a ZIP archive or JPEG). In other words, after a file attachment is compressed, it cannot be further compressed by network compression. This is an important point to remember when determining the results of network compression at your site. If you run a test in which you mail a large number of compressed attachments, the resulting gains produced by network compression might appear smaller compared to mailing documents with uncompressed attachments. (We further illustrate this fact in the test results that follow.) In Domino, you can enable network compression using the Notes client, the Domino Administrator, or both. From Notes: 1. Open Notes, and choose File → Preferences → User Preferences. 2. Go to the Ports tab. 3. Select the port on which you want to enable network compression from the list. 4. Select Compress network data to enable network compression for the selected port on the local Notes client. (This does not affect the server.) From the Domino Administrator: 1. In the Server pane, click the server on which you want to enable network compression. 2. Go to the Server → Status tabs, and then click Server Tasks. 3. Under Tools on the right, click Server → Setup Ports. 4. Select the port on which to turn on network compression. (Make sure Port enabled is selected for the port.) 5. Select Compress network data. 6. Click OK. 7. Restart the ports on the server so that network compression takes effect. This enables network compression for the selected ports on this server. (This does not affect the Notes client.)<br />
<br />
824<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
23.4.11 Using port encryption You can encrypt network data on specific ports to prevent the network eavesdropping possible with a network protocol analyzer. Network encryption occurs at the network transfer layer of a selected protocol and is independent of other forms of encryption. Network data is encrypted only while it is in transit. When the data has been received and stored, network encryption is no longer in effect. Network data encryption occurs if you enable network data encryption on either side of a network connection. For example, if you enable encryption on a TCP/IP port on a server, you do not need to enable encryption on the TCP/IP ports on workstations or servers that connect to the server. Multiple high-speed encrypted connections to a server can affect server performance. Encrypting network data has little effect on client performance. In addition, encrypted network data cannot be compressed. Therefore, if you encrypt network data on a port that uses a data-compressing modem, you do not gain the throughput benefits from the modem. In testing port encryption versus no port encryption, the relative processor utilization increases in the 5% to 10% range. Also, user response is slightly affected. However, if you are running the server in the less than 60% processor utilization range, the user’s response time and performance degradation will not be significant. Note: Network compression in Domino will not yield benefits with bandwidth savings if the data is encrypted (such as SSL for Domino Web Access).<br />
<br />
23.4.12 Lotus Domino partitioning Using Domino server partitioning, you can run multiple instances of the Domino server on a single computer. By doing so, you reduce hardware expenses and minimize the number of computers to administer because, instead of purchasing multiple small computers to run Domino servers that might not take advantage of the resources available to them, you can purchase a single, more powerful computer and run multiple instances of the Domino server on that single machine. On a Domino partitioned server, all partitions share the same Domino program directory, and thus share one set of Domino executable files. However, each partition has its own Domino data directory and NOTES.INI file. Thus, each has its own copy of the Domino Directory and other administrative databases.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
825<br />
<br />
If one partition shuts down, the others continue to run. If a partition encounters a fatal error, the Domino fault recovery feature restarts only that partition, not the entire computer. Use Lotus Domino partitioning technology to increase scaling of active user loads and to use more powerful configurations such as faster clock cycles, fibre-connected I/O subsystems, operating system kernel to CPU binding, and multiple I/O controllers. The Server_Max_Concurrent_trans notes.ini setting sets the limit for the number of concurrently scheduled transactions on a server. If you use this setting to set the maximum number of concurrent transactions on partitioned servers, Lotus recommends that the sum of the limits be 20 transactions or less. For example, if you are running four partitioned servers on a computer, you would set the limit for each partitioned server at five transactions. The syntax of the statement is as follows: Server_Max_Concurrent_Trans = [number]<br />
<br />
23.4.13 Balancing workload in a Lotus Domino cluster If you are going to scale Domino to higher user loads, you might need to use Domino clustering. Clustering divides the load among the servers so that you can take advantage of high-end hardware configurations. Domino clusters provide scalability. It is easy to add servers to a cluster and to remove them from a cluster as the workload changes. You can also balance the workload among servers. This important feature lets the system distribute workload at the same time demand fluctuates or in the event of a server failure. Workload balancing lets the clustering technology distribute the workload dynamically, based on performance thresholds set in each server.<br />
<br />
Server availability threshold The server availability threshold specifies the lowest acceptable server availability index. By setting this percentage value for each server in a cluster, you determine how the workload is distributed among cluster members. Approximately once each minute, Domino computes the server availability index and compares it to the server availability threshold that you set. If the availability index is less than the availability threshold, the server is marked as BUSY. When a server is marked as BUSY, requests to open databases are redirected to another server, if one is available. When the availability index becomes higher than the availability threshold again, the BUSY condition is withdrawn.<br />
<br />
826<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To set the server availability threshold, add the following line to the notes.ini file: Server_Availability_Threshold = [number] The higher the number that you enter, the less workload the server can carry before going into the BUSY state. Entering the number 100 automatically puts the server into the BUSY state, regardless of its actual availability. Entering the number 0 disables workload balancing for that server. The default value is 0. The server availability threshold is a key configuration setting for Lotus Domino cluster workload balancing. Setting the threshold too high can cause user requests to fail unnecessarily. Setting the threshold too low can result in poor performance for some users who might have received better service from another server. You should also consider how a server outage might affect server workload. For the failover capability, you might want to set the server availability threshold to allow some extra capacity to handle the failover workload.<br />
<br />
Network traffic considerations Cluster replication is constantly updating replicated databases over your network. These are not scheduled replication events; they are real-time updates to the databases. Cluster replication transactions are primarily small record updates, but you might be concerned about potential congestion on your network. In practice, many Domino cluster installations have found out that the network overhead is actually quite small. However, you do have the option of isolating cluster replication traffic by setting up a private LAN dedicated to intra-cluster communication between servers. The additional expenditure required to do this is very small; all you need is an additional network adapter in each server, a hub, and wiring. A parameter in each server’s notes.ini file, Server_Cluster_Default_Port, specifies which port is used for intra-cluster network traffic.<br />
<br />
23.5 Maintaining Lotus Domino servers Lotus Domino provides a wide range of tools for gathering information about a server and for diagnosing and fixing problems. Include the following maintenance tasks in your administration plan: Back up the server regularly. This is especially important for database recovery. Check for dead mail. Use the Administration Process to remove users’ names globally from access control lists (ACLs).<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
827<br />
<br />
If necessary, run the Database Fixup utility manually to fix corrupted databases and prevent server crashes. Check the log file for problems with replication, mail routing, scheduling, database integrity, and communication links. Monitor server statistics and events to track disk space, server load, memory and swap space, and to head off server trouble. Monitor database replication and ACL changes using event monitors. Periodically run the Compact utility to keep wasted database free space to a minimum. Use performance monitoring tools, such as the Windows Performance console, to measure current server performance. Refer to Chapter 14, “Windows tools” on page 471 for more information.<br />
<br />
23.6 Planning for future growth The scalability, or capacity for expansion, of your system depends on how well you plan for growth. For example, if you initially deploy your system with just a few servers and decide on a peer-to-peer topology for server communication, you might need to rethink your topology as you add servers. Additionally, you might need to upgrade your network and computing infrastructure as you add servers and users. If you find yourself in this situation, consider deploying a Domino server cluster as a way to achieve scalability with the added advantages of server and database availability.<br />
<br />
23.7 Top 10 ways to improve Domino performance Lotus Domino server performance is not determined purely by the power of your hardware. Other factors can have a significant effect on the overall system performance. Making sure your network design, replication schedules, and database distribution are not causing inefficiencies are just three examples. In the following list, we have tried to put items in order of potential improvement, highest first, but that might not be the case for your specific environment. Also,<br />
<br />
828<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
clustering can provide significant improvement, but appears toward the end of the list because it is not usually implemented for performance alone. 1. Enable transaction logging The easiest way to improve disk performance is to enable transaction logging. Always place the log files on a separate disk device, preferably running RAID-1. Lotus recommends archive logging. 2. Verify your memory configuration After the disk subsystem, memory is a potential bottleneck. Verify that you have installed enough memory to support the number of users on your server. If required, add more memory. The suggested minimum is 0.2 MB per active user. You might want to reserve more memory for more advanced users performing more complicated tasks, perhaps 1 MB for each concurrent user. Also reserve memory for any additional server task, such as Directory Assistance or Indexing. 3. Distribute your databases over multiple physical disk devices You can greatly improve disk utilization by spreading the disk workload across multiple physical disk devices or RAID arrays. Configure several disk devices and create directory links pointing to these devices using the Domino Administrator tool. Spread databases and user mail files roughly evenly across the disks. 4. Use RAID-1 disk subsystems For performance coupled with fault tolerance, RAID-1 is superior to both RAID-0 and RAID-5. Configure one RAID-1 array for the operating system, paging file, and Domino program file, another array for transaction logging, and one or more arrays for Domino data. Configure more disks per RAID array to get better disk read/write data throughput. Use a stripe size of 32 KB. This is the optimal stripe unit size for Domino on Windows 2003 (R2) servers. Span disk devices across multiple SCSI channels (when large numbers of disks are being used). 5. Improve your network design If the network is highly utilized, split your network into smaller segments. Connect your critical servers directly to a network switch. Upgrade your network adapters from 10 Mbps to 100 Mbps or faster adapters. If even better network performance is required, consider ATM.<br />
<br />
Chapter 23. Lotus Domino<br />
<br />
829<br />
<br />
6. Use Network compression You could implement network compression to reduce network traffic between servers and between clients and servers. 7. Increase the number of mail.box files on your mail servers Mail routing can become a bottleneck in your mail servers. You can increase the number of mail.box databases to ease the congestion. 8. Improve your Domino design The Domino network and system design is at the heart of every system. Consider at least the following: – Plan replication schedules carefully. Consider hub and spoke design if there are several Domino servers in your network. – Dedicate Domino servers to roles: one server for mail, one for Remote Access Service (RAS), another for databases. 9. Implement Domino clustering Connect your servers into a cluster to achieve better availability, scalability and workload balancing. 10.Upgrade to faster CPUs Add more CPUs to improve raw processing power. The amount of L2 cache is very important to server CPU performance. A larger L2 cache is always better. Remember, the only sure way to know if one of your server resources is becoming a bottleneck is to monitor the system on an ongoing basis. Use the Windows Performance console to collect data. Analyze it to learn how to correct the possible performance bottlenecks. Also use Domino’s own tools in the Statistics & Events database.<br />
<br />
830<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
24<br />
<br />
Chapter 24.<br />
<br />
Microsoft Exchange Server Exchange Server1 is the Microsoft solution for e-mail and collaboration applications. Exchange Server only runs in Active Directory environments, because it prepares the Active Directory database to store data such as user, groups, or configuration settings. This chapter describes basic tuning actions for Microsoft Exchange Server 2003 and includes the following topics: <br />
<br />
24.1, “Planning guidelines” on page 832 24.2, “Tuning guidelines for subsystems” on page 834 24.3, “Exchange Server 2003 operations” on page 843 24.4, “Exchange Server 2003 downloads” on page 844 24.5, “Exchange 2007” on page 845<br />
<br />
Exchange Server is not only an application for sending and receiving e-mails, it also manages mobile and Web-based access to information, shared calenders and tasks, and can be integrated in Microsoft Share Point Services. The current version of Microsoft Exchange Server is 2003 SP2. Similar to Exchange Server 2003 (R2), it has many compatibility modes to allow administrators to migrate an existing Exchange 5.5 or 2000 environment to 2003 slowly.<br />
<br />
1<br />
<br />
Product screen shots and content reprinted with permission from Microsoft Corporation<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
831<br />
<br />
There are two versions of Microsoft Exchange Server 2003: Standard Edition, which supports one message database per server SP1 supports databases up to 16 GB size SP2 supports databases up to 75 GB size Enterprise Edition, which supports up to 20 databases per server SP1 and SP2 supports databases up to 8 TB size Exchange Server 2003 runs on 32-bit Windows Server 2003 or Windows Server 2000 SP4 servers, but it is recommended to use Windows Server 2003, because it adds some enhancements and functionality to an Exchange Server 2003 infrastructure. There is no support for 64-bit. The next generation Exchange 2007 will only run on 64-bit versions of Windows. An Exchange Server 2003 infrastructure is very complex. If you install the applications, it is designed to be self tuning. However, depending on your environment, you have to do some improvements manually. In this chapter, we provide some suggestions to optimize the performance of Exchange Server 2003, but it is impossible to cover all possible scenarios because of the complex infrastructure of the product. You can find extensive information published by Microsoft in the Exchange 2003 Technical Documentation Library, which is available online at: http://www.microsoft.com/technet/prodtechnol/exchange/2003/library Alternatively, you can also find information if you join the community at: http://www.microsoft.com/exchange/community<br />
<br />
24.1 Planning guidelines In most cases, implementing Exchange in a company infrastructure is a large project. For a new Active Directory Domain, you should integrate Exchange in the planning phase. If the integration is into a domain or a migration update, you should also review and perhaps modify your infrastructure to meet the demands that Exchange needs.<br />
<br />
832<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
You should first review the network infrastructure. Depending on your sites, the number of users, and your LAN and WAN connection, you should decide if you want to use a centrally or a distributed Exchange design. You can reduce the number of servers that you need by investing in high-end servers, which brings the following advantages: <br />
<br />
Centralizing the hardware Centralizing the administration Reducing the number of servers Reducing the total cost of ownership.<br />
<br />
Investing in a high-end server will not help, however, if your LAN and WAN connections cannot handle all the traffic generated by the server. If you migrate an existing Exchange environment, you can find useful information in determining performance issues by using the Microsoft Exchange Server Best Practices Analyzer Tool, which is available online at: http://www.microsoft.com/downloads/details.aspx?familyid=dbab201f-4bee4943-ac22-e2ddbd258df3&displaylang=en This tool helps you to identify configuration problems proactively by creating a report about your Exchange infrastructure. This report includes critical information about configuration issues and potential problems. In the Exchange infrastructure, subsystems other than the network can also be a bottleneck. Even when there is a good physical site structure, the right Active Directory domain and forest partitioning as well as the domain controller and global catalog server placement can effect network performance and can cause long response times or unnecessary replication data. You can find a planning guide for designing an Active Directory domain at: http://go.microsoft.com/fwlink/?linkid=18348 Note: If you edit configuration settings on your Exchange server, it takes a while until the new changes are replicated to the whole forest or until a new sized buffer is working correctly. So, do not attempt to compare performance measurements immediately after you make changes. Depending on your configuration and the changes that you make, it can take hours, days, or even longer until the whole forest is using the new configuration settings.<br />
<br />
Chapter 24. Microsoft Exchange Server<br />
<br />
833<br />
<br />
24.2 Tuning guidelines for subsystems Mail servers provide the messaging needs for a company. Storing and delivering messages or information are the main purposes of the mail server. If there are multiple sites in your infrastructure, you might need to use connectors between your mail servers. As described in 2.6, “E-mail servers” on page 18, the subsystems that are most likely to be sources of bottlenecks in a e-mail server are: Memory CPU Disk Refer to Chapter 19, “Analyzing bottlenecks for servers running Windows” on page 655 for more information about detecting and removing bottlenecks that are related to these subsystems. As you identify which subsystems are potential sources of bottlenecks, you will have a general idea of what initial actions to take regarding performance optimization. This section provides some basic tuning guidelines for optimal performance.<br />
<br />
24.2.1 Network subsystem Contrary to popular belief, the network subsystem is rarely the cause for performance bottlenecks in Exchange operations. While sufficient bandwidth is needed to support the user base, testing has shown that even the workload generated by several thousands of users will not saturate a Gigabit Ethernet connection. You should take care that you have the latest device drivers and BIOS and Firmware releases installed. With Exchange, it is important that you have fast access to the Active Directory. Exchange stores data in the Active Directory, and so the Exchange server often has to query data from the Domain Controllers, especially from the Global Catalog owners. These requests only cause small packets, but there are a lot of them. So, every Exchange server should have a Global Catalog at its site. Alternatively, if there are not many users at the remote site, you need a fast and constant WAN connection so that the Exchange server does not suffer from too long a response time, which can decrease performance.<br />
<br />
834<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
A network adapter with TCP Offload Engine (TOE) or I/O Acceleration Technology (IOAT), described in 10.3, “Advanced network features” on page 259, improves performance because they unburden the CPU. If you plan to use IPSec to encrypt network traffic, there is also a specialized network adapter available to offload the encryption process from the main CPU. Tip: Use the /3GB switch in boot.ini, as described in 24.2.2, “Memory subsystem” on page 835. This switch can increase performance in some cases. If you increase the JET cache from 512 to 1 GB, you can store more AD Objects in the memory, which can cause quicker response times.<br />
<br />
24.2.2 Memory subsystem Having sufficient RAM is very important to an Exchange server’s performance. If the system does not have enough memory to support Exchange database buffers and Exchange services, the operating system has to provide memory using a paging file. However, using a paging file significantly affects performance. You should check memory usage using Performance Monitor Counters, as described in 19.3, “Analyzing memory bottlenecks” on page 661, and if necessary, add more RAM to increase the database buffer size.<br />
<br />
Setting the /3GB switch Regardless off the amount of physical memory in your system, Windows addresses a virtual address space of 4 GB and allocates 50% of this space to the kernel-mode processes (for example, the operating system and the kernel-mode drivers). Windows allocates the other 2 GB to the user-mode processes (for example applications). However, Windows supports the use of a startup switch that lets you change the allocation use of memory, on systems that have 1 GB or more physical memory installed, as shown in Figure 24-1.<br />
<br />
Chapter 24. Microsoft Exchange Server<br />
<br />
835<br />
<br />
0GB<br />
<br />
0GB<br />
<br />
User Mode User Mode 2GB<br />
<br />
3GB<br />
<br />
Kernel Kernel 4GB<br />
<br />
4GB Without /3GB<br />
<br />
With /3GB<br />
<br />
Figure 24-1 Affect of the /3GB Switch<br />
<br />
The /3GB switch is supported in the following Windows Server 2003 versions: Windows Server 2003, Standard Edition Windows Server 2003, Enterprise Edition Windows Server 2003, Datacenter Edition To set the /3GB switch on a Windows Server 2003 operating system, do the following: 1. Right-click My Computer and select Properties. 2. Go to the Advanced Tab. 3. Click Startup and Recovery Settings. 4. Click Edit in the System Startup Area 5. In the [Operating Systems] section, add the /3GB switch to the end of the line, as follows: multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise" /fastdetect /3GB 6. Save the changes and close notepad. 7. Close all dialog boxes by clicking OK. 8. Restart the server.<br />
<br />
836<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Setting the /userva switch The /userva switch allows you to enter more system page table entries on the server, but you should only set the switch in combination with an application that supports it, such as Exchange. The /userva switch provides a better division of the virtual address space. You should use this switch only in combination with the /3GB switch. By default, the switch is set to zero (0). Exchange supports the switch to 3030. You can add more users without consuming all available system resources. To set the /userva switch, after you add the /3GB, add the /userva=3030 switch, as follows: multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise" /fastdetect /3GB /USERVA=3030 Note: Microsoft does not recommend that you set the switches in the following cases: For Exchange servers that do not include any mailboxes or public folders, such as mail gateways. For Exchange servers that are at the same time Active Directory or Global Catalog servers. Microsoft recommends that you have separated Active Directory and Exchange servers. For a Microsoft Small Business Server installed as an Domain Controller and Exchange server.<br />
<br />
Setting the HeapDecomitFreeBlockThreshold key If you have more than 1 GB of physical RAM installed in your Exchange server, we recommend that you edit this registry key. A heap is a reserved area of memory. If you set this key, it forces the Heap Manager to free up memory. Freeing memory reduces memory fragmentation. To set the HeapDecomitFreeBlockThreshold value, follow these steps: 1. Run regedit. 2. Navigate to HKLM\SYSTEM\CurrentControlSet\Control\Session Manager 3. Set the HeapDeCommitFreeBlockThreshold value to 40000 (Figure 24-2). 4. Click OK. 5. Restart the server.<br />
<br />
Chapter 24. Microsoft Exchange Server<br />
<br />
837<br />
<br />
Figure 24-2 Setting the Heap Value<br />
<br />
Optimizing your store database cache size (ESE Buffer) The Extensible Storage Engine (ESE) Buffer provides a large caching area for database transactions before they are committed to the store. By default, if the /3GB switch is set, Exchange Server 2003 allocates 896 MB of RAM for this buffer. If the disk performance is not optimal or if the server is heavily loaded, it can increase performance to set a larger ESE Buffer. Depending on your configuration, you might have to decrease or increase the size of the buffer. If you run other applications on the same server as the Exchange application, it might be useful to reduce the ESE Buffer size to increase the overall performance from all applications on this server. If you want to increase the buffer, we recommend that you to monitor your server before you do so using the Performance Monitor as described in 19.3, “Analyzing memory bottlenecks” on page 661. When using the Performance Monitor, you should analyze the following objects and counters: Performance Object: Process Performance Counter: Virtual Bytes Instance: STORE You should also gather the information under a normal workload. The value that is seen in the Performance Utility should be for a server with the /3GB switch set at a value less than 2.5 GB. If the switch is not set, it should be less than 1.8 GB. You have to compare the gathered data with these values. If they are less, there is no need to change the switch. If they are near this value, you can add additional space for the ESE Buffer, up to 1200 MB. Use caution when setting this value, because in some cases, setting this switch increases performance, but on a very large mailbox server, a too large buffer size affects the stability of the operating system. So, monitor performance first, and take care if you decide to set the switch.<br />
<br />
838<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Editing the ESE Buffer size The msExchESEParamCacheSizeMax value sets the ESE Buffer size. Its value is expressed as a page count, and must be set to an exact multiple of 8192 for maximum efficiency: <br />
<br />
Servers with the /3GB switch set: default size = 229,376 (896 MB) Servers without the /3GB switch set: default size = 147,456 (576 MB) Maximum value with the /3GB switch set = 311,296 (1.2 GB) Maximum value without the /3GB switch set =196 608 (768 MB)<br />
<br />
To change this value, do the following: 1. Insert the Server 2003 CD-ROM. 2. Navigate to the Support\Tools Folder. 3. Install the Active Directory Services Interfaces (ADSI) tool. 4. Start the ADSI tool. 5. Under Configuration Container expand CN=servername, DC=domainname, DC=root hint. 6. Expand CN=Services, CN=Microsoft Exchange. 7. Expand CN=OrganizationName where OrganizationName is the name of your organization. 8. Expand CN=Administrative Groups. 9. Expand CN=First Administrative Group where First Administrative Group is the name of your administrative group. 10.Expand CN=Servers, CN=servername. 11.Under CN=servername, right-click CN=InformationStore, and then click Properties. 12.In the Select which properties to view list, click Both. 13.In the Select a property to view list, click msExchESEParamCacheSizeMax. 14.In the Edit Attribute box, type the value that you want to assign this attribute. 15.Click Set and then click OK. 16.Quit the ADSI Edit utility. 17.Allow for sufficient time for this value to be replicated throughout the Active Directory forest. 18.Restart the Microsoft Exchange Information Store service on the Exchange server.<br />
<br />
Chapter 24. Microsoft Exchange Server<br />
<br />
839<br />
<br />
Additional improvement Do not forget to optimize the server for data throughput for network applications as described in 11.6, “File system cache” on page 309 and shown in Figure 24-3. With this setting, you can optimize how Windows Server 2003 prioritizes memory allocations and thread priorities for network services versus local applications.<br />
<br />
Figure 24-3 Maximize data throughput for network applications<br />
<br />
24.2.3 CPU Exchange can also cause a lot of utilization on your server CPU. There are some functions included in Exchange, such as the search service or antivirus scanning, that can burden the processor. Exchange scales very well with multi-core systems but can only effectively use up to eight cores. For a server with more than eight cores, use partitioning to split the server in multiple 8-core or 4-core servers. As an alternative, you can also set the process affinity of the store.exe process to only eight cores. The optimal number of processors to scale an Exchange server depends on the role that the server plays. For example, while a backend mailbox server that hosts many MAPI connections might make efficient use of an 8-core server, a<br />
<br />
840<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Exchange Web Access server might makes better use of a 4-core server. To plan your requirements, Microsoft has information about how to calculate mailbox server CPU requirements available at: http://www.microsoft.com/technet/prodtechnol/exchange/guides/E2k3Perf_ ScalGuide/ccec1c70-a3df-404a-93e0-b74077e3d013.mspx?mfr=true If you run a single Exchange server that handles all services and the CPU is a bottleneck, you should switch to a frontend and backend design. It is possible to separate the servers and run client access protocols such as HTTP, POP3, or IMAP4 from the servers that maintain the actual information stores. The frontend server accepts protocol requests from a client system, interprets them, makes additional lookups to the directory service, and then relays the requests to the backend server. This method offloads the CPU-intensive tasks of handling client requests from the Information Store server, allowing more scalable deployments. Because little data is stored on the local system, frontend server performance depends mostly on the server’s CPU rather than on disk or memory subsystems.<br />
<br />
24.2.4 Disk subsystem The attached storage to an Exchange server is very important because disk subsystem bottlenecks can cause very grave performance issues. The most common problem is that people tend to plan storage designs for capacity and not for performance. When you plan an Exchange server, you should calculate the following volumes at a minimum: RAID-1 for the operating system RAID-1 for the transaction log files RAID-10 for the database store However, this basic configuration is not enough. The most important factor that can affect disk subsystem issues is the throughput and the latency of your read and write operations. An simple and effective rule is that it is better to have more disks with a smaller capacity than having fewer disks with a larger capacity, because if you stripe the arrays over more disks, you have a greater number of spindles which increases performance. This is one of the most important investments you can make in your disk subsystem. See 9.6.2, “Number of drives” on page 202 for more information. If you want to calculate your disk I/O requirements, see the following for more information: http://www.microsoft.com/technet/prodtechnol/exchange/guides/StoragePer formance/fa839f7d-f876-42c4-a335-338a1eb04d89.mspx?mfr=true<br />
<br />
Chapter 24. Microsoft Exchange Server<br />
<br />
841<br />
<br />
After you have optimized your hardware, you should organize the storage subsystem to optimize performance. The following are recommendations about how to place your data files: Database files Place all database files on a single volume that is only used for these files. To increase performance, the disks where the database files are stored should have fast random access speeds. (RAID-10) Content Indexing Files The default location for the indexing files is on the same disk as the page file. Moving the indexing files to a separate volume increases performance. If your disk subsystem performs well enough, you can place these files on the database file volume, because the indexing file is also a random access file. (RAID-10) Single Instance Storage Mailboxes that belong to the same workgroups and distribution lists should be stored in the same database. (RAID-10) Transaction Log Files The most important performance factor for Transaction Log Files is the latency. So, they should be placed in a single storage group. (RAID-10) SMTP queue The SMTP queue should be placed in a own volume that is distributed over multiple disk spindles. The used spindles should not handle other volumes. (RAID-10) Page file The default location for the page file is on the operating system volume. Move the page file to a own volume that is distributed over multiple spindles improves performance. (RAID-1) MTA queue The MTA queue (Message Transfer Agent) should never reside on a log or database volume. The best way is to provide a separate set of spindles for the SMTP and MTA queue together. (RAID-10) To improve performance, it is recommended to connect your system volume and your page file through Direct Attached Storage (DAS). The other volumes should be placed on a Storage Area Network (SAN).<br />
<br />
842<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
After you have calculated, installed, and optimized your Disk Subsystem, there is a tool to verify the performance expectations prior than in a production environment available at: http://www.microsoft.com/technet/prodtechnol/exchange/guides/ StoragePerformance/fa839f7d-f876-42c4-a335-338a1eb04d89.mspx?mfr=true<br />
<br />
24.3 Exchange Server 2003 operations Apart from the subsystem specific guidelines, there are some general issues that can impact Exchange Server 2003 performance. These issues can be grouped as: Information Store maintenance Backup and restore Coordinating these task plays a significant part in maintaining server performance, because they can generate heavy workload, especially on the disk subsystem.<br />
<br />
Information Store maintenance Online maintenance is a very important factor for Exchange Server 2003 performance. It keeps mailboxes and public stores in good health by executing three main tasks: Checks AD for any deleted mailboxes Expired message and mailbox cleanup Exchange Server 2003 does not purge data from its Information Store right away when a user deletes a message. This enables users to recover messages that were deleted accidently without the need to restore them from backups. The data is kept for a period of time defined by the system administrator and is only purged after the retention period has expired. The same concept applies to data that is stored in user mailboxes should a system administrator choose to delete them. Performs online database defragmentation As messages are added to and deleted from the Information Store databases, the database files start to fragment very similarly to file systems. Over time, this fragmentation slows down data access within the database because data is spread throughout different parts of the database file. Also, the database files will continue to grow unnecessarily because the space that is freed up by deletions (white space) cannot be reused effectively. To compensate for this, defragmentation reorganizes the internal structure of the<br />
<br />
Chapter 24. Microsoft Exchange Server<br />
<br />
843<br />
<br />
database. The white space is recovered, thus speeding up data access and reducing database file size. By default, Exchange performs these housekeeping tasks between 1 a.m. and 5 a.m. each day, but you can schedule the maintenance individually for each database on the server. For servers that need to be available to users 24 hours a day, it might be advisable to spread out or to stagger the maintenance periods over the day. Information Store maintenance can have a severe impact on Exchange performance, depending on the amount of data that is stored on the server. You can find more information about the Information Store maintenance at: http://blogs.msdn.com/jeremyk/archive/2004/06/12/154283.aspx<br />
<br />
Backup and restore Online database backup plays a critical part in Exchange operations and must be coordinated carefully with Information Store maintenance to minimize the impact on server performance. You should create separate backup jobs for each storage group and attune its runtime with they maintenance jobs for the databases within that group. Only one backup process can run on each storage group at a time. Also, running a backup halts Information Store maintenance on all databases within the storage group. Therefore, you need to schedule your backup jobs so that they only start after Information Store maintenance has finished. If a backup starts while maintenance is running, the maintenance job resumes after the backup job has finished.<br />
<br />
24.4 Exchange Server 2003 downloads A need for every Exchange administrator is the Microsoft Downloads for Exchange 2003, which is available at: http://www.microsoft.com/technet/prodtechnol/exchange/downloads/2003/ At this site, you can find service packs and trial software as well as a link to the Tools for Exchange Server 2003 Web site. This site offers very useful tools to enhance your Exchange environment.<br />
<br />
844<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
24.5 Exchange 2007 Exchange 2007 will only be supported on 64-bit platforms. Thus, with Exchange 2007, Exchange will effectively use 16 GB of RAM and beyond. Microsoft published a summary of the hardware that they recommend for reuse at: http://www.microsoft.com/technet/prodtechnol/exchange/2003/articles/ e2k3-e12hardware.mspx<br />
<br />
Chapter 24. Microsoft Exchange Server<br />
<br />
845<br />
<br />
846<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
25<br />
<br />
Chapter 25.<br />
<br />
IBM DB2 Universal Database IBM DB2 Universal Database (DB2 UDB) is the relational database system for building robust, scalable, and reliable data management applications. From smaller department needs to enterprise wide transaction and data warehouse systems, DB2 UDB scales to meet the most demanding business requirements. Support and tight integration for both Windows and Linux running on Intel architectures is a strong feature of DB2 UDB development. DB2 UDB performance tuning can be extremely simple or more demanding depending upon the requirements and scale of the data management solution. DB2 UDB also provides a number of wizards for configuring performance and other database objects. This chapter is a quick overview of DB2 UDB performance tuning. DB2 performance tuning can be segmented into the following areas: 1. 2. 3. 4.<br />
<br />
Memory Disk CPU Networking<br />
<br />
For more information about tuning DB2 UDB, see DB2 UDB V8.2 on the Windows Environment, SG24-7102.<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
847<br />
<br />
In addition, the DB2 UDB product publications also include the book Administration Guide: Performance, which is available from: http://www.ibm.com/software/data/db2/udb/support/manualsv8.html In this chapter, we discuss the following topics: <br />
<br />
25.1, “Optimizing the operating system” on page 848 25.2, “CPU subsystem” on page 849 25.3, “Memory subsystem” on page 851 25.4, “Disk subsystem” on page 855 25.5, “Networking and miscellaneous topics” on page 858<br />
<br />
25.1 Optimizing the operating system You can improve the performance of your DB2 UDB server by changing various operating system settings for Windows Server 2003 and Linux. We recommend that you implement these changes after you have installed DB2. Review the following chapters for details: Chapter 11, “Microsoft Windows Server” on page 295 Chapter 12, “Linux” on page 371 You need to configure the operating system to service the unique demands of an application server. For Windows, the following settings are relevant: Configure priority to background services, as described in 11.2, “Windows Server 2003, 64-bit (x64) Editions” on page 298 Configure the file system cache to maximize throughput for network applications, as described in 11.6, “File system cache” on page 309 For servers with more than 4 GB, add the /PAE to BOOT.INI, as described in 11.14, “Using PAE and AWE to access memory above 4 GB (32-bit x86)” on page 335. For servers with more than 4 GB but less than 16 GB, also add the /3GB parameter to BOOT.INI, as described in 11.13, “The /3GB BOOT.INI parameter (32-bit x86)” on page 334.<br />
<br />
848<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
25.2 CPU subsystem DB2 UDB makes very effective use of SMP hardware. The degree to which individual queries can use multiple processors (intra-partition parallelism) can be influenced. The database manager configuration parameter INTRA_PARALLEL controls whether or not intra-partition parallelism is enabled. Although there are many tweaks that you could use to optimize this type of parallelism, the general rule is to enable intra-partition parallelism if the typical database workload is for complex queries with relatively few users (for example, DSS, OLAP, and data warehouse systems). For SQL workloads that are simple, repetitive, require a large number of users or queries (for example, OLTP systems), then intra-partition parallelism should be disabled. If intra-partition parallelism is disabled, then the DEFAULT_DEGREE database configuration parameter should be set to 1 to avoid unnecessary overhead. Disabling intra-partition parallelism will not limit the number of processors utilized by DB2 UDB but only the number of processors available for individual SQL statements.<br />
<br />
25.2.1 Logical nodes DB2's shared-nothing architecture makes it possible for data that is partitioned across multiple nodes in a cluster to be processed collectively to produce a single query result. In a shared-nothing cluster, the data on one node in a cluster environment cannot be seen or processed by processors located in another node. To satisfy a query against a shared-nothing database, DB2 divides that query into parts that can be executed on each node in parallel. When the results from the query are returned to the managing node in the cluster, they are aggregated and returned as a single result to the user. This type of partitioned database is ideally suited scalable design of NUMA systems such as the x445. DB2 is designed to manage data locally and minimize traffic between the nodes in a cluster. Defining a logical cluster environment on a single SMP x445 maximizes the server's performance and scalability by minimizing the traffic between scalable nodes in the server. This approach uses the optimized design of the x445 4-way SMP scalable node, while minimizing latencies that are incurred by accessing data that is local to another scalable node in the system. When a DB2 database is partitioned, each database partition is referred to as a logical node. When defining clustered databases on the x445, consider defining one logical node per logical processor in the system. Thus, on an 8-way x445<br />
<br />
Chapter 25. IBM DB2 Universal Database<br />
<br />
849<br />
<br />
with Hyper-Threading enabled, define 16 logical nodes. Each logical node of a DB2 partitioned database is run as a separate process with its own address space. On Windows, the address space is 2 GB, or 3 GB is the /3GB parameter is specified in BOOT.INI (see 25.1, “Optimizing the operating system” on page 848).<br />
<br />
25.2.2 Hyper-Threading Enabling Hyper-Threading in the BIOS means the operating system (provided the operating system supports Hyper-Threading) and applications running on it to see each physical processor as a pair of logical processors. These additional logical processors increased the overall system processing capability allowing two threads to run concurrently on a single physical processor. For implementations such as Business Intelligence, enabling Hyper-Threading can provide a performance improvement. Lab tests have seen upwards of a 13% improvement in an 8-way configuration.<br />
<br />
25.2.3 Processor affinity When DB2 is implemented on NUMA-based systems such as the x445, it is important to localize resources to each of the scalable nodes. When the database is set up correctly, it is important to ensure that DB2 assigns a logical node (LN) to run on the processor that has a direct physical connection to data belonging to that logical node of the database. Without correct affinity settings, processors might routinely access data owned by processors in other scalable nodes, creating increased, unnecessary traffic and contention across the system. In a DB2 multiple logical node environment, ensure that a LN is affinitized to a single physical processor and its corresponding logical processor. A simple table scan against data owned by a logical node in the system, in conjunction with the performance tab in Task Manager can be used to verify that processor affinity is set correctly. Example 25-1 illustrates the setting of processor affinity. See the DB2 UDB documentation for additional information about setting processor affinity. Tip: When Hyper-Threading is enabled on the 8-way x445, processor numbers 0 through 7 correspond to physical processors, processor numbers 8 through 15 correspond to logical processors.<br />
<br />
850<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Example 25-1 Script to set processor affinity<br />
<br />
rem Set affinities for LN0 hyperthreading enabled. db2set db2processors=4,12 db2set db2processors=5,13 db2set db2processors=6,14 db2set db2processors=7,15 db2set db2processors=0,8 db2set db2processors=1,9 db2set db2processors=2,10 db2set db2processors=3,11<br />
<br />
-> LN7 -i -i -i -i -i -i -i -i<br />
<br />
for 8 processor system with<br />
<br />
hrinst hrinst hrinst hrinst hrinst hrinst hrinst hrinst<br />
<br />
0 1 2 3 4 5 6 7<br />
<br />
25.2.4 Metrics to watch Useful counters to monitor with respect to the processor subsystem are: Processor: %Processor Time If the Windows Performance Monitor counter, %Processor Time is consistently higher than 70% to 80%, then there might be a CPU bottleneck. This counter value might indicate that your system is running close to capacity, or it might point to another hidden issue. Package Cache Hit Ratio (%) The DB2 Performance counter, Package Cache Hit Ratio (%) can be a very significant indicator of poor CPU utilization. If the hit ratio is not in the high 90s, and increasing the Package Cache does not improve the hit percentage, then the likely problem is an application design flaw. This type of design flaw can result in 90% of CPU utilization being wasted on repetitive query compilation. If SQL statements in an application are repeated often and the only changes are the variables used, it is important to utilize the programming technique of parameter markers. This will decrease by an order of magnitude the amount of time spent creating access plans for the SQL statements. Refer to the Application Development Guide that is included with the online documentation for further reference.<br />
<br />
25.3 Memory subsystem Proper memory utilization and data buffering is the key to database performance. All database activities at some point will require utilization of the various DB2 UDB buffer pools, caches, and heaps. The primary memory area is the buffer pool which is the work area for DB2 data pages. A data page is the allocation unit<br />
<br />
Chapter 25. IBM DB2 Universal Database<br />
<br />
851<br />
<br />
where rows of table or index data are stored. The purpose of the buffer pool is to improve system performance. Data can be accessed much faster from memory than from disk; therefore, the fewer times the database manager needs to read from or write to a disk (I/O), the better the performance. Key ideas for effective buffer pool utilization are: The larger the buffer pool, the more data can be stored Keep more frequently accessed data in the buffer pool Keep essential index data in the buffer pool for faster data access To accomplish these goals, changes to the size and number of buffer pools might be required. You can accomplish these changes manually, or DB2 UDB provides some very helpful wizards to help with this process. The DB2 Database Performance Wizard is an excellent starting point for system-wide tuning. To start the wizard, select the desired database in the DB2 Control Center, and click Selected → Configure Performance Using Wizard. By providing the wizard with information about your system, transaction workload, and database requirements, it will produce logical defaults for most DB2 UDB parameters. Figure 25-1 shows the first window produced by the wizard.<br />
<br />
Figure 25-1 DB2 Database Performance Wizard<br />
<br />
852<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
As you proceed through each window by providing requested information, DB2 creates a list of recommended changes.<br />
<br />
Figure 25-2 DB2 Database Performance Wizard<br />
<br />
Figure 25-2 shows the user requesting information about the amount of memory that can be dedicated to the database. You can segment your data into different categories by creating separate table spaces for disparate data (frequently accessed, history, index, sequentially accessed data, randomly accessed data, index data, LOB data). By segmenting the data, you can assign different buffer pools to corresponding table spaces, therefore controlling the data and system memory utilization. For more information about table spaces, see 25.4, “Disk subsystem” on page 855. The buffer pool can also be set manually. For business intelligence environments with multiple logical nodes, and 2 GB of physical memory per logical node, start with a buffer pool of roughly half the application address space (2 GB or 3 GB for 32-bit platforms). Note that the buffer pools for each logical node are separate, and unique physical memory should be available for each logical node. DB2 also gives you the flexibility to define buffer pool page size. The page size chosen for the buffer pool must equal the page size of the tablespace that<br />
<br />
Chapter 25. IBM DB2 Universal Database<br />
<br />
853<br />
<br />
includes database data and index pages that will cached there. Buffer pool page size is defined at the time the buffer pool is created. The code fragment in Example 25-2 illustrates creation of a buffer pool. The parameter size -1 is used to indicate the size of the bufferpool will be determined by the database configuration parameter BUFFPAGE. Example 25-2 Creating a buffer pool and specifying the page size<br />
<br />
create bufferpool BP16K size -1 pagesize 16k; The desire to buffer data should never be compromised by over-allocating buffers, which results in memory swapping and increased I/O. One DB2 profile registry variable DB2NTNOCACHE allows prevention of system caching of DB2 UDB data files in Windows environments. This allows more memory to be used for DB2 UDB buffer pools without duplicating similar efforts by the operating system. Another important parameter to consider is sort heap, which is the number of 4 KB pages in memory to use for sorting. Memory used for sort heap is separate from memory used for the buffer pool. DB2 configuration parameters are set at an instance or database level. Database manager configuration parameters are created at the instance level and are the same for all databases within the instance, while database configuration parameters are unique to a database. Buffer pool size and sort heap size are examples of database configuration parameters. When multiple logical nodes are defined, database configuration parameters must be set for each node of a database, and should be set consistently across the system. The command sequence in Example 25-3 illustrates the setting of each of these parameters. See the DB2 UDB product documentation for a complete listing of DB2 configuration parameters. Example 25-3 Setting the sort heap and buffer pool<br />
<br />
Db2start Db2 connect to HR Db2 update db config for HRDB using BUFFPAGE 60000 Db2 update db config for HRDB using SORTHEAP 6400 Db2 connect reset Db2stop<br />
<br />
854<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
25.3.1 Metrics to watch By selecting the desired database in the DB2 Control Center and choosing Performance Monitoring from the Selected menu, you can access the DB2 Performance Monitor. A useful counter to monitor is the Buffer Pool Hit Ratio (%). This DB2 Performance Monitor metric should be in the 90% range for effective buffer pool utilization. If this number is significantly below 90%, consider adding memory and increasing the size of the buffer pool, or better, utilizing prefetching as discussed in 25.4, “Disk subsystem” on page 855.<br />
<br />
25.4 Disk subsystem As mentioned, one goal of performance tuning is to minimize disk I/O. If I/O is necessary, it is important to make it as efficient as possible. There are two effective concepts for efficient I/O: Prefetching, the concept of moving data into the buffer pool before it is required by the application. When to do prefetching is largely a function of the database engine either determining it will be beneficial to prefetch data or, as a query is performed, detecting that prefetching will be helpful. Parallel I/O, the movement of data more quickly into the buffer pool by performing I/O operations simultaneously rather than sequentially There are a number of ways to affect the amount of parallel I/O. The overall principle is to try to spread data access across as many physical drives as possible. RAID devices perform this task at a lower layer than the database engine. DB2 UDB can perform parallel I/O on a single RAID volume or across RAID volumes. If data is placed on a single RAID volume (for example, drive D:) the database engine does not know that the device is capable of performing multiple I/O operations simultaneously. The DB2 profile registry variable DB2_PARALLEL_IO is used to inform the database engine that volumes are available for parallel I/O operations. To set this variable, open a DB2 command window and enter the command: DB2SET DB2_PARALLEL_IO=* This will turn on parallel I/O for all volumes. As a general rule, this is a good idea when using hardware RAID devices. If database data is placed on multiple RAID volumes, they are automatically available for parallelism.<br />
<br />
Chapter 25. IBM DB2 Universal Database<br />
<br />
855<br />
<br />
25.4.1 Table spaces Data storage in DB2 UDB is based upon the concept of table spaces. A table space is created from one or more containers. Containers are the locations for data placement and can be directories, specific files, or entire volumes. There are two types of table spaces: System Managed Space (SMS) SMS table spaces are the simplest to administer. They include one container, which is a directory where DB2 UDB will create and manipulate data files as needed and is limited only by the size of the volume where the directory lives. This type of table space, however, cannot send table and index data pages into separate buffer pools. Also, data pages might not be contiguous because the OS has greater control of physical placement of data on the volume. Database Managed Space (DMS) DMS table spaces have greater control of data placement. The containers for a DMS table space are either files of specified size or entire raw volumes. If using file containers, there should only be one file per volume and each container should be of the same size. As DMS table space containers fill up, you can either increase the size of the containers if the containing volumes have available space, or you can add containers to the table space. The DMS table space type allows for table and index data to be separated into different table spaces and therefore separate buffer pools. DMS data is also more likely to be stored contiguously, making for more efficient I/O. The database configuration parameter NUM_IOSERVERS specifies how many database agents are available for performing prefetching and parallel I/O operations. This should be set to one or two more than the number of physical drives that make up the volumes where DB2 data is stored. Another important I/O operation is logging. Because all database data changes must be logged in order to guarantee data consistency, it is important that the logging activity does not become a bottleneck. The DB2 UDB database logs should be placed on a volume with enough physical drives to meet the write intensive work of the logger. The database configuration parameter NEWLOGPATH is used to specify the path where the database logs are created.<br />
<br />
856<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
25.4.2 Page size, extent size, and prefetch size Defining a table space requires setting three key parameters that directly affect performance of the I/O subsystem: Page size Extent size Prefetch size DB2 stores data on a page and for Business Intelligence workloads for example, a large page size of 16 KB is recommended. In DB2, the smallest allocation unit of space is known as an extent. The extent size is defined as the number of contiguous pages to be allocated at one time. DB2 allocates an extent in each database container in a round-robin fashion to ensure that data is balanced across the available disk subsystem. DB2 allows the user to define the extent size. The extent size chosen for a Business Intelligence environment, for example, is based on the following formula: size × Disks in RAID array Extent size = Stripe --------------------------------------------------------------------------------Buffer pool page size KB × 6 Extent size = 64 ------------------------- = 24 16 KB<br />
<br />
With a stripe size is 64 KB, a six-drive RAID array, and a 16 KB buffer pool page size, the extent size can be calculated to be 24. Thus, DB2 allocates 24 16 KB pages to each database container in a table space before moving to the next one. The last parameter is prefetch size, which should be set to a multiple of the extent size multiplied by the number of containers in the table space. For example if you have two containers in the table space (per logical node), the prefetch size is 24x2=48. This means that each time DB2 executes a prefetch operation, it reads 48x 16 KB pages from disk and store them in memory. Because data stored in this logical node of the database is striped across two containers, DB2 will be able to prefetch 48x 16 KB pages as two parallel 24x16 KB or 384 KB I/O operations. Note that 384 KB divided by 64 KB equals 6, which is the number of drives defined in each of our disk arrays. So during a prefetch operation, DB2 will issue one I/O, which will result in reading 64 KB of data from each of 6 disk drives simultaneously. Not only is the necessary data moved into memory before it is needed, it is moved in efficiently with a high degree of parallelism. This reduces unnecessary latency, which lengthens response time. As DB2 processes database data, it<br />
<br />
Chapter 25. IBM DB2 Universal Database<br />
<br />
857<br />
<br />
should do fewer physical I/Os because data will have already been prefetched into memory.<br />
<br />
25.4.3 Metrics to watch For the disk subsystem, useful counters to monitor are: Total Synchronous I/O Time (ms) This DB2 Performance Monitor metric is the total amount of time applications had to wait on an I/O operation, read or write from disk. If the data pages required were in the buffer pool or were placed in the buffer pool by an asynchronous operation (prefetch), then a synchronous I/O would not be needed. Ensure that this time is reasonable, as measured by user response time (that is, how long users of the system are willing to wait for their transaction to be completed), compared to the amount of users and workload. Physical Disk Queue Length This Windows performance counter can also be helpful. If Physical Disk Queue Length is larger than 2 for any monitored drive (multiply by the number of physical disks for RAID arrays), then you might have an I/O bottleneck. Try to maximize I/O across as many physical drives as possible.<br />
<br />
25.5 Networking and miscellaneous topics The DB2 UDB optimizer determines how to implement submitted SQL statements. It is vitally important that the optimizer have current statistics on tables and indexes to build effective access plans. The RUNSTATS command is used to update the statistics of tables and indexes in the system catalogs. It is available either from the DB2 Control Center or from the DB2 Command Line. The command should be run when a significant amount of data has been changed or when indexes are created. The command can be run on individual tables; one convenient method to gather statistics on all tables is to make use of the REORGCHK utility. Network performance tuning is often overlooked in database performance tuning. The two basic ideas are to limit the number of records and limit the number of requests that flow from the server to the remote client application. Configuring the database server can only partially do this. If database requests are for large numbers of rows but only a small subset of these rows is being processed, then if possible, this should be altered to retrieve smaller subsets or only the necessary rows.<br />
<br />
858<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
The client-side parameter RQRIOBLK controls the size of the buffer used to store retrieved rows on the client. If a single SQL statement retrieves large data (for example, large objects) or the number or size of rows being transferred is large, increasing this parameter will allow more data to be stored on the client for processing. This will reduce the frequency of requests to the database for additional data. However, this increases the overhead on the database server for each connection and should be weighed accordingly. DB2 performance tuning need not be overly complicated. Under most circumstances, avoiding a few key roadblocks and using the provided DB2 wizards will greatly enhance your system performance.<br />
<br />
Chapter 25. IBM DB2 Universal Database<br />
<br />
859<br />
<br />
860<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
26<br />
<br />
Chapter 26.<br />
<br />
Microsoft SQL Server Microsoft SQL Server1 provides a powerful database platform for delivering critical business applications efficiently. SQL Server builds on the power, scalability, and manageability of the Microsoft Windows 2000 Server and Windows Server 2003 operating systems to provide the reliability and advanced capabilities of high-end, client/server database management. This chapter focuses on SQL Server 2000 and SQL Server 2005 and discusses the factors that affect SQL Server performance, including monitoring, tuning tools, and processes. SQL Server tuning involves: 1. Understanding the application environment. 2. Customizing the hardware configuration to support the application requirements. 3. Customizing the database design and configuration to best use system resources. 4. Tuning the underlying operating system to get the best performance from the hardware configuration.<br />
<br />
1<br />
<br />
Product screen captures and content reprinted with permission from Microsoft Corporation.<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
861<br />
<br />
26.1 Features of Microsoft SQL Server 2000 Microsoft SQL Server 2000 has evolved significantly into a largely self-tuning database system. Many of the settings that required manual tuning on SQL Server 7.0 function automatically with more recent versions of SQL Server. Improved features of SQL Server 2000 include: Configuration options SQL Server is largely self-configuring, self-tuning, and self-managing. LazyWriter and read-ahead manager are self-tuning. The max async I/O option is likely the only sp_configure option that you need to configure initially when dealing with servers with larger amounts of storage. This reduction in tuning requirements saves valuable administrative time. You can still configure and adjust many of the sp_configure options manually that were available in previous versions of SQL Server. Nonetheless, it is recommended that database administrators allow SQL Server to configure and to tune all sp_configure options automatically for which defaults are provided. This automatic configuration and tuning allows SQL Server to monitor and to adjust continually the configuration of the database system as factors affecting it change. This might include such factors as memory and processor utilization of SQL Server itself and other applications or services running on the server. LazyWriter tuning SQL Server configures and tunes the LazyWriter automatically. There is no longer the need to tune Free Buffer and LazyWriter I/O activity manually. You can still monitor these with the SQL Server Performance Monitor objects as described in 26.11, “SQL Server performance objects” on page 916. Checkpoint tuning In previous versions of SQL Server, the recovery interval setting was used to tune the checkpoint process. SQL Server now tunes the recovery interval setting automatically. The SQL Server default of zero for recovery interval indicates that SQL Server takes responsibility for monitoring and tuning the recovery interval parameter automatically. This default setting normally maintains recovery times less than 60 seconds for all databases as long as there are no exceptionally long-running transactions present on the system. Log manager Previous versions of SQL Server shared RAM cache with data pages and manual tuning of the log manager was sometimes necessary. Now, SQL Server tunes the performance of SQL Server log manager automatically. The log manager manages its own log cache, and there is no longer the<br />
<br />
862<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
dependency on the SYSLOGS table that existed in earlier versions of SQL Server. This separating of log file management from data cache management brings enhanced performance for both components. SQL Server log manager is also capable of performing disk I/O in larger byte sizes than before. The larger I/O size combined with the sequential nature of SQL Server logging help to make disk I/O performance very good for the log manager. Federated servers Federated servers allows SQL Server tables and views to be spread out across multiple servers, enabling SQL Server to scale to very large database workloads. 64 GB memory support SQL Server 2000 SP4 supports up to 64 GB of RAM using AWE with Windows Server 2003 RC2, Enterprise or DataCenter Edition (32-bit and 64-bit). SQL Server 2000 added x64 support with SP4.<br />
<br />
26.1.1 SQL Server 2000 editions SQL Server 2000 is available in several different versions. These are: <br />
<br />
SQL Server 2000, Standard Edition SQL Server 2000, Enterprise Edition SQL Server 2000, Developer Edition SQL Server 2000, Personal Edition SQL Server 2000, Desktop Engine (also known as MSDE)<br />
<br />
Only the Standard and Enterprise editions of SQL Server 2000 can be deployed in a live, production server environment. You cannot deploy the other editions of SQL Server 2000 to live systems due to both technical and licensing restrictions. You can install the Standard and Enterprise editions only on Windows server operating systems, such as Windows 2000 Server or Windows Server 2003. You cannot install them on desktop operating systems, such as Windows 2000 Professional or Windows XP. Thus, enterprise users of SQL Server 2000 typically only have one decision to make: Standard Edition or Enterprise Edition.<br />
<br />
Features of SQL Server 2000, Enterprise Edition SQL Server 2000, Enterprise Edition is the most feature-rich version in the SQL Server suite. It is the choice for mission-critical deployments of SQL Server that require the highest levels of performance and scalability. Enterprise Edition offers greater memory and processor support, failover clustering, and advanced analysis tools. It can support up to 32 processor threads in the 32-bit version and<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
863<br />
<br />
64 processor threads in the 64-bit version. Failover clustering is available for up to four nodes in the 32-bit edition and eight nodes in the 64-bit edition. Enterprise Edition is the choice for demanding data warehousing and online transactional processing (OLAP) requirements, including support for OLAP partitions and real-time processing and updating of very large OLAP cubes. Note: Many of these highest levels of hardware scalability are only available when teamed with the Enterprise or Datacenter editions of Windows Server 2003.<br />
<br />
Features of SQL Server 2000, Standard Edition SQL Server 2000, Standard Edition is a more affordable version that is suitable for servers that do not require the extended hardware support and availability features of Enterprise Edition. Standard Edition, available in a 32-bit edition only, can support up to 2 GB of memory and four processors. It does not support clustering nor the advanced OLAP handling features of Enterprise Edition, but most other features available in Enterprise Edition are available in Standard Edition.<br />
<br />
26.2 Features of SQL Server 2005 SQL Server 2005 includes the following enhancements: A new user-mode operating system, called SQLOS, which is NUMA aware (see 26.2.5, “SQL Server Operating System” on page 869) Major enhancements to the Database Engine, Analysis Services, Integration Services (formerly Data Transformation Services), and Reporting Services A new client layer called SQL Client New tools for management and development New features for high availability, such as Database Mirroring, Data Partitioning, and support for hot-add memory<br />
<br />
864<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
26.2.1 SQL Server 2005 editions There are five editions of SQL Server 2005: <br />
<br />
SQL Server 2005 Enterprise Edition (32-bit and 64-bit) SQL Server 2005 Standard Edition (32-bit and 64-bit) SQL Server 2005 Workgroup Edition (32-bit only) SQL Server 2005 Express Edition (32-bit only) SQL Server 2005 Developer Edition (32-bit and 64-bit)<br />
<br />
Greater use of server resources and product features is possible with these systems, starting from the Express Edition up through Workgroup, Standard, and the Enterprise Editions. The Developer Edition includes all the features of the Enterprise Edition but is licensed for development and testing, not for production use. The Enterprise Edition scales to the performance levels that are required to support the largest enterprise online transaction processing (OLTP), highly complex data analysis, data warehousing systems, and Web sites. Enterprise Edition has comprehensive business intelligence and analytical capabilities and high availability features such as failover clustering and database mirroring that allow it to handle the most mission-critical enterprise workloads. Enterprise Edition is the most comprehensive edition of SQL Server 2005 and is ideal for the largest organizations and the most complex requirements. It is also available in a 120-day Evaluation Edition for the 32-bit or 64-bit platform. For a detailed description of which features are supported in which edition, see the SQL Server 2005 Books Online topic, “Features Supported by the Editions of SQL Server 2005.” Official Microsoft SQL Server 2005 documentation is available online at: http://www.microsoft.com/sql/2005<br />
<br />
26.2.2 New and enhanced features of SQL Server 2005 In this section, we provide an overview of each of the major enhancements in SQL Server 2005.<br />
<br />
Database Engine enhancements The Database Engine introduces new programmability enhancements such as integration with the Microsoft .NET Framework (specifically, the Common Language Runtime component) and Transact-SQL enhancements, new XML functionality, and new data types. It also includes improvements to the scalability and availability of databases.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
865<br />
<br />
The new features provided by the Database Engine include: Database mirroring You can use database mirroring to enhance the availability of SQL Server 2005 databases by providing fast failover and automatic client redirection to a secondary server. In contrast to failover clustering, database mirroring keeps two copies of the database, does not require specialized hardware, and is easier to set up and maintain. Data partitioning Partitioning tables and indexes provide the following benefits: – Large tables or indexes can be more manageable because of quick and efficient access to or management of data subsets, while maintaining the integrity of the overall collection. – Querying large tables or indexes is likely to be faster and more efficient on multiple CPU computers. Hot-add memory Additional physical memory can be installed in a running server, and SQL Server 2005 will recognize and use the additional memory immediately. For example, suppose you are running SQL Server 2005 and Windows Server 2003, Enterprise Edition 32-bit on a computer with 16 GB of physical memory. The 32-bit operating system is configured to limit applications to 2 GB of virtual memory address space. AWE has been activated on SQL Server and the -h switch enabled during startup. To increase server performance, you add another 16 GB of memory. SQL Server recognizes the additional memory immediately and begins to use it as necessary, without you restarting of the server. Note: Removing physical memory from the system still requires that you restart the server. Online restore With SQL Server 2005, database administrators can perform a restore operation while the rest of the database remains online and available. Online restore improves the availability of SQL Server because only the data being restored is unavailable. Online indexing operations The online index option allows concurrent modifications (updates, deletes, and inserts) to the table or any associated indexes during index maintenance operations. For example, while a clustered index is being rebuilt, users can<br />
<br />
866<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
continue to make updates to the underlying data and perform queries against the data. Fast recovery A new faster recovery option improves the availability of SQL Server databases. Users can reconnect to a recovering database after the transaction log has been rolled forward. It is no longer necessary to wait for the rollback phase to complete. Security enhancements SQL Server 2005 includes security enhancements such as database encryption, secure default settings, password policy enforcement, fine grained permissions control, and an enhanced security model. Dedicated administrator connection SQL Server 2005 introduces a dedicated administrator connection (DAC) that administrators can use to access a running server even if the server is locked or otherwise unavailable. This capability allows administrators to troubleshoot problems on a server by executing diagnostic functions or Transact-SQL statements. Snapshot isolation SQL Server 2005 introduces a new snapshot isolation level that is intended to enhance concurrency for OLTP applications. In earlier versions of SQL Server, concurrency was based solely on locking, which can cause blocking and deadlocking problems for some applications. Snapshot isolation depends on enhancements to row versioning and is intended to improve performance by avoiding reader-writer blocking scenarios.<br />
<br />
26.2.3 Analysis Services enhancements Analysis Services introduces new management tools, an integrated development environment, and integration with the .NET Framework. Many new features extend the data mining and analysis capabilities of Analysis Services. New or improved features that are provided by the Analysis Services include: <br />
<br />
New data mining algorithms Clustering support Key performance indicators (KPIs) Relational online analytical processing (ROLAP) Proactive caching Integration with Microsoft Office products<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
867<br />
<br />
26.2.4 Additional enhancements and features The following enhancements expand on existing functionality, or in the case of Notification Services and Server Broker, provide new functionality. Integration Services enhancements Integration Services (formerly Data Transformation Services) introduces a new extensible architecture and a new designer that separates job flow from data flow and provides a rich set of control flow semantics. Integration Services also provides improvements to package management and deployment, along with many new packaged tasks and transformations. The new Maintenance Plan Wizard builds packages that you can customize with Integration Services. Replication enhancements Replication offers improvements in security, manageability, availability, programmability, mobility, scalability, and performance, such as: – – – – –<br />
<br />
A new Replication Monitor The ability to make schema changes to published tables Improved support for non-SQL Server Subscribers Merge synchronization over the Web Relaxed large data type restriction on Updatable Transactional Subscribers<br />
<br />
Reporting Services enhancements Reporting Services is a new server-based reporting platform that supports report authoring, distribution, management, and user access. Tools and utilities enhancements SQL Server 2005 introduces an integrated suite of management and development tools that improve the ease-of-use, manageability, and operations support for large scale SQL Server systems. Data access interfaces enhancements SQL Server 2005 supplies improvements in Microsoft Data Access Components (MDAC) and the .NET Frameworks SQL Client provider for greater ease-of-use, control, and productivity for developers of database applications. Notification Services Notification Services is a new platform for building highly-scaled applications that send and receive notifications. Notification Services can send timely, personalized messages to thousands or millions of subscribers using a wide variety of devices<br />
<br />
868<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Service Broker Service Broker is a new technology for building database-intensive applications that are secure, reliable, and scalable. Service Broker provides message queues that the applications use to communicate requests and responses.<br />
<br />
26.2.5 SQL Server Operating System New to SQL Server 2005 is a layer between SQL Server and the operating system called SQL Server Operating System (SQLOS). SQLOS is a user-level, highly configurable operating system with a powerful API that enables automatic locality and advanced parallelism. SQLOS attempts to hide the complexity of the underlying hardware from high-level programmers. It also provides a comprehensive set of features to programmers who are willing to take advantage of the hardware underneath the system. SQLOS services include non-preemptive scheduling, memory management, deadlock detection, exception handling, hosting for external components such as CLR, and other services. SQLOS has a hierarchical architecture, which means that SQLOS changes its structure based on the hardware platform on which SQL Server 2005 is running. Figure 26-1 illustrates SQLOS on an SMP-based server.<br />
<br />
SQLOS<br />
<br />
SMP<br />
<br />
SOS_MemoryNode<br />
<br />
SOS_CPUNode<br />
<br />
SOS_Scheduler<br />
<br />
SOS_Scheduler<br />
<br />
SOS_Task<br />
<br />
SOS_Task<br />
<br />
SOS_Scheduler<br />
<br />
SOS_CPUNode SOS_Task<br />
<br />
SOS_Scheduler<br />
<br />
SOS_Task<br />
<br />
Figure 26-1 SQLOS for an SMP server (8-socket)<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
869<br />
<br />
Figure 26-2 illustrates SQLOS on a 2-node NUMA server such as a 2-node (8-socket) x3950.<br />
<br />
NUMA<br />
<br />
SQLOS<br />
<br />
SOS_MemoryNode<br />
<br />
SOS_MemoryNode<br />
<br />
SOS_CPUNode<br />
<br />
SOS_CPUNode<br />
<br />
SOS_Scheduler<br />
<br />
SOS_Scheduler<br />
<br />
SOS_Task<br />
<br />
SOS_Task<br />
<br />
SOS_Scheduler<br />
<br />
SOS_CPUNode SOS_Task<br />
<br />
SOS_Scheduler<br />
<br />
SOS_Task<br />
<br />
Figure 26-2 SQLOS on a NUMA server (2-nodes)<br />
<br />
Notice that with SQLOS on an SMP-based server, SQLOS only has one memory node, while SQLOS on the NUMA server two memory nodes. SQLOS recognizes the two nodes and the information of locality regarding processors and memory in the server.<br />
<br />
26.3 Choosing which version of Windows to use Each new release of the Windows server operating system offers performance improvements over previous versions. While many performance advantages were notable between Windows NT 4.0 Server and Windows 2000 Server, the performance benefits between Windows 2000 Server and Windows Server 2003 are even more significant. Wherever possible, for best performance gains, we recommend that you use SQL Server 2005 teamed with the Windows Server 2003 operating system.<br />
<br />
870<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Some of the notable improvements in the Windows Server 2003 operating system that are particularly appealing to SQL Server implementations include the following: Improved support for Intel Hyper-Threading processors. Clustering support for up to eight nodes. Considerable enhancements have been made to disk I/O performance. Simultaneously, CPU resources needed to service I/O requests has dropped. The implementation of the TCP/IP protocol has received a significant performance boost over previous versions. SMP scalability to 32-way symmetric multi-processing support with the 32-bit edition of Windows Server 2003, Datacenter Edition, and 64-way SMP with the 64-bit edition of Windows Server 2003, Datacenter Edition. Memory scalability to 64 GB of RAM on SQL Server 2000, 128 GB of RAM on SQL Server 2005 with the 32-bit edition of Windows Server 2003, Datacenter Edition right up to a massive 1024 GB of RAM with the x64 edition of Windows Server 2003, Datacenter Edition and SQL Server 2005.<br />
<br />
26.3.1 Windows and SQL Server, both 32-bit When you run 32-bit Windows Server 2003, you can run either 32-bit SQL Server 2000 or 32-bit SQL Server 2005. All processes run in 32-bit, so there is no 64-bit option for SQL Server in this case. SQL Server 2000 runs as a user process, with the standard 4 GB of address space (see Figure 26-3 on page 872), normally divided into two: 2 GB for the kernel 2 GB for the user mode portion With 4 GB of RAM, you can use the /3GB boot.ini switch to change the split to 1 GB for the kernel and 3 GB for the user portion of the virtual address space. If you have more than 4 GB of physical memory installed, 32-bit SQL Server can use it as the database buffer pool. However, you must enable Physical Address Extension (PAE) in Windows (add the /PAE switch to boot.ini) and enable Address Windowing Extensions (AWE) in SQL Server (using sp_configure). All SQL Server memory objects, binary code, data buffers, database page headers (512 MB for 64 GB of database buffers), sort area, connections, stored procedure caches, open cursors (basically, everything but the unmapped cached database pages) must fit in the 2 GB user mode portion of the process address space. To access the database buffer pool pages above the 4 GB line, SQL Server must map them into the address space below the 4 GB line. Refer to Figure 26-3. This mapping incurs some performance overhead.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
871<br />
<br />
Note: For systems with more than 16 GB of RAM, you cannot enable both /3GB and /PAE. See 11.13, “The /3GB BOOT.INI parameter (32-bit x86)” on page 334. With AWE on, SQL Server 2000 allocates its full max server memory (MB) amount and never releases memory until it is shut down.<br />
<br />
64 GB physical memory 4 GB virtual address space 2 GB kernal VA space<br />
<br />
AWE mapping<br />
<br />
2 GB user VA space SQL Server cached db pages 2 GB<br />
<br />
SQL Server VA space 4 to 64 GB<br />
<br />
Binary code Data buffers DB page headers Sort area Connections Stored proc cache Open cursors<br />
<br />
2 to 4 GB 0 to 2 GB<br />
<br />
0 GB<br />
<br />
Figure 26-3 SQL Server 2000 uses AWE memory only for buffer pool pages<br />
<br />
If your SQL Server workload is constrained by the 2 GB user mode limit for reasons other than requiring more database buffer pages, then it will not benefit from having more than 4 GB of physical memory. This type of bottleneck is a hardware architectural bottleneck, which is relieved by using 64-bit hardware and software. SQL Server 2005 32-bit experiences the same hardware architectural bottleneck. SQL Server 2005 manages all of its memory (including the AWE memory) dynamically, releasing and allocating memory in response to internal and external memory pressure.<br />
<br />
872<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
26.3.2 Windows 64-bit and SQL Server 32-bit When you have Windows Server 2003, x64 Edition installed, you can run either SQL Server 2000 with Service Pack 4 or 32-bit SQL Server 2005. SQL Server 2000 will run in WOW64 (Windows on Windows) in a user mode address space of 4 GB. Unlike running the same application on 32-bit Windows, SQL Server 2000 has a full 4 GB of user mode address space because the 64-bit kernel runs in its own address space. This provides some relief for SQL Server workloads that are constrained by the 2 GB user mode address space under a 32-bit operating system. SQL Server 2000 can also be configured to use AWE, which allows it to access up to 64 GB of physical memory on EM64T processors. The AWE memory above 4 GB must still be mapped into the lower 4 GB user mode address space to be used. SQL Server 2005 32-bit benefits in the same way that SQL Server 2000 does with a 4 GB user mode address space that is not shared with the operating system. With AWE enabled, SQL Server 2005 can access up to 128 GB of physical memory. SQL Server 2005 manages all of its memory (including the AWE memory) dynamically, releasing and allocating memory in response to internal and external memory pressure.<br />
<br />
26.3.3 Windows and SQL Server 2005, both 64-bit When you run Windows Server 2003 x64, you can (and should) install the x64 version of SQL Server 2005. There is no 64-bit version of SQL Server 2000 for use on EM64T-based servers. SQL Server 2005 x64 enjoys the same memory addressability as the 64-bit operating system. The 64-bit user mode address space is not limited to 4 GB, and it can use memory up to the operating system maximum for any purpose, not just for database buffers. AWE mapping is not required because the memory model is flat under 64-bit addressing. This provides the most efficient utilization of resources for SQL Server 2005. For more information about 32-bit and 64-bit memory addressing with Windows Server 20, see Introducing Windows Server x64 on IBM Eserver xSeries Servers, REDP-3982.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
873<br />
<br />
26.4 The database environment Effective SQL Server optimization involves a careful analysis of areas where potential changes could yield the largest performance improvement given different situations. SQL Server database performance deficiencies are usually caused by the logical database design and configuration, index design, and query design areas. While other system-level performance issues, such as memory, cache buffers, hardware, and so forth, are certainly candidates worthy of investigation, experience shows that the performance gain from these areas is often only of an incremental nature. SQL Server performs automatic self-tuning by managing whatever hardware resources are available. This reduces the need for hardware subsystems tuning beyond initial correct sizing and implementation. In SQL Server performance tuning, the following questions should first be asked to understand your system environment: Is this a dedicated database server? How many users will access the database concurrently? Are users accessing the database directly or through a Web (or other) front-end? What type of I/O is being generated by the application: random or sequential? What are the network requirements of the application? What hardware resources are available? With the answers to these questions, you will be able to plan your tuning approach more successfully. Database layout design should take into consideration the effect on SQL Server performance. You can do the following to optimize database performance: Distribute the database evenly across available disk drives and arrays. Separate randomly accessed data from sequentially accessed data, such as the transaction log. Create indexes on database tables, where necessary, to improve database access and overall responsiveness.<br />
<br />
874<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
26.5 SQL Server performance tuning basics To achieve the best performance results, consider the following principles of SQL Server performance tuning: SQL Server is a self-tuning database server. SQL Server can automatically configure and self-tune databases to provide maximum database performance even for enterprise-level database servers. SQL Server dynamically configures and tunes system settings to provide peak performance for variable user type and workload. The most important hardware factor typically affecting database performance is physical memory. RAM is primarily used as a database server buffer cache by SQL Server. If there is not enough physical memory available, data is paged to disk, slowing down the server. The driving focus of database performance tuning is to reduce disk I/O so that buffer cache is best utilized. Adequate RAM in a system will reduce disk I/O from paging and speed system performance. Good throughput of the disk subsystem is essential to provide sufficient I/O capacity for the database server to run without the queuing of I/O requests to disk. Disk queuing results will impact performance negatively and will increase I/O response time. When SQL Server does read-ahead for table or index scans, it will perform up to 1 MB reads. Generally it is optimal to configure disk arrays for approximately a 512 KB stripe size. Use indexes to reduce the number of I/O operations. To minimize I/O requests produced by queries, good indexes must be create and maintained. When SQL Server supports a large number of user connections, client application and index architecture are an important performance factor. Client applications determine the load on server and incorrect index architecture will generate increased I/O operations. Application developers should understand SQL Server architectural basics and how to take full advantage of SQL Server indexes to minimize I/O operations. SQL Server provides the Profiler and Index Tuning Wizard to analyze and change index architecture. The SQL Server Profiler can be used to log SQL Server activity which can then be submitted to the Index Tuning Wizard to make index changes to improve performance. Regular use of these tools helps SQL Server performance as the overall query workload changes over time. SQL Server provides a revised set of Performance console objects and counters, which are designed to provide helpful information for monitoring and analyzing the operations of SQL Server. Use SQL Server Query Analyzer and Graphical ShowPlan to analyze problematic SQL queries.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
875<br />
<br />
26.6 Server subsystems You should consider each of the main server subsystems (memory, disk, processor, and network), how they impact SQL Server performance, and then tune them accordingly.<br />
<br />
26.6.1 Memory The most critical performance factor for SQL Server is the memory subsystem. In most cases, the more physical RAM a server has, the better SQL Server will perform. Adding memory to a server is often a relatively inexpensive and easy way to boost performance. System memory is primarily used for database buffer cache by SQL Server. Access to data in the system memory is much faster than access to the same information from disk and having enough memory means that disk I/O and paging will be reduced, offering considerable performance improvements.<br />
<br />
Memory configuration In almost all instances, memory optimization settings are best left to SQL Server itself. SQL Server knows how to most efficiently allocate server RAM, meaning no administrator intervention is required. SQL Server 2000 with SP4 is NUMA-aware as well as SQL Server 2005. Prior to SP4, SQL 2000 is not NUMA-aware. When self-tuning, SQL Server does not hold onto RAM it is not using. If it does not need additional RAM, it releases it to the operating system. Similarly, when set to self-tune, SQL Server uses RAM that is available to it when it requires it. While the minimum and maximum amounts of RAM that SQL Server can use are able to be configured manually, these are normally best left to self-tune dynamically. Only in some instances, such as where a server is not dedicated to SQL Server, and memory sharing needs to be managed, should these settings be changed from the default. As with all changes, the impact to SQL Server and overall system performance should be monitored to see if performance gains have been realized through adjusting values from the default. Of particular note is the System Monitor metric SQL Server:Buffer Manager:Buffer Cache Hit Ratio. The goal in memory tuning should be to see this setting become as close to 100% as possible. The closer that this setting is to 100%, the more efficiently SQL Server will use system memory.<br />
<br />
876<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
You can access the memory configuration panel for SQL Server 2000 by following these steps: 1. Open SQL Server Enterprise Manager. 2. In the left frame, expand Microsoft SQL Servers and locate the SQL Server Group for the server that you want to configure. 3. Right-click the SQL Server in the right-frame and click Properties. 4. Select the Memory tab (Figure 26-4).<br />
<br />
The default settings for dynamic memory management will be best for almost all SQL Server implementations.<br />
<br />
Figure 26-4 SQL Server 2000 memory usage configuration<br />
<br />
In Figure 26-4, the default Minimum memory setting is 0 MB. In most cases, you need to leave this value as it is. Increasing the amount of memory ensures that the amount of memory is available for SQL Server, but removes it from other possibly system requirements. To change this setting, move the slide-bar to the right.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
877<br />
<br />
You can access the memory configuration panel for SQL Server 2005 by following these steps: 1. Select Start → Programs → Microsoft SQL Server 2005 → SQL Server Management Studio. 2. Connect to SQL Server instance. 3. Right-click the instance icon in Object Explore, and select Properties. 4. Select Memory in Select a page in the left pane. 5. Change the Minimum server memory and Maximum server memory (Figure 26-5).<br />
<br />
Figure 26-5 Memory setting in SQL Server 2005 Management Studio<br />
<br />
878<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
The default Maximum memory setting is the total amount of physical RAM available to the server in MB. The minimum setting to which this can be reduced is 4 MB, which is the absolute least amount that SQL Server needs to function. It is best to leave this setting at the default maximum. However, if you need to confirm memory as available for other applications on the server and to prevent it from being allocated to SQL Server, move the slide-bar to the left to reduce the amount of RAM available to SQL Server. If the maximum amount of memory available to SQL Server is reduced below an optimal setting, this will almost certainly causing system paging, which can seriously impede system performance. Using the Use a fixed memory size setting removes any self-tuning capabilities of SQL Server for the use of system memory. While this setting will allow an administrator to be certain of how much memory SQL Server uses, it does not allow SQL Server to self-tune and optimize memory usage as it sees fit. The Reserve physical memory for SQL Server setting ensures that physical RAM is allocated for the instance of SQL Server equal to the amount set in the above memory control. This means that this physical will be dedicated to SQL Server processes and that SQL Server memory pages will never be swapped out to disk. This offers a potentially considerable performance gain however it comes at the expense of all other processes running on the system not being able to use this server RAM. The Minimum query memory setting specifies the minimum amount of memory that SQL server will allocate for per user for query execution. Note this setting is measured in kilobytes, not megabytes. The default is 1024 KB (1 MB).<br />
<br />
Detecting bottlenecks The best way to monitor memory usage is to use this Windows performance counter: SQLServer:BufferNode. For a server that is running only SQL Server, there should be no paging. That is,<br />
<br />
Memory:Pages/sec should be zero. Most of the memory that SQL Server uses is allocated for the Buffer Pool, which consists of 8 KB pages. Buffer Manager:Database pages gives the number of buffer pages that include database content. Take the number of buffer pages, multiply it by 8 KB, and compare it to the total amount of physical memory in the server. This gives you an indication of how much of your physical memory is actually being used. Use DBCC MemoryStatus to obtain a snapshot of how memory is allocated on each of the NUMA nodes and how it is consumed by the pools and caches in SQL Server.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
879<br />
<br />
Memory usage in SQL Server 2000 and 2005 is very complex. However, there is a simple procedure for determining whether a workload will benefit from additional memory. It is based on the observation that most applications respond to increased memory according to the general graph shown in Figure 26-6.<br />
<br />
Transactions/sec<br />
<br />
Adding or subtracting memory makes a little change in performance<br />
<br />
Adding or subtracting memory makes a big change in performance<br />
<br />
Amount of physical memory Figure 26-6 How more memory affects performance<br />
<br />
The procedure measures the application performance with two amounts of memory—your current amount of memory and a small amount less. If the difference in performance is not great, then that means you are at the top part of the curve and adding more memory does not improve performance much. On the other hand, if the difference in performance is great, then you are at the bottom part of the curve and adding memory is likely to improve performance, up to the point where the curve begins to flatten out. Refer to 19.3, “Analyzing memory bottlenecks” on page 661 for more information about detecting and removing memory-related bottlenecks.<br />
<br />
880<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
26.6.2 Disk Because the primary function of a database server is to store and retrieve data, the disk subsystem is a critical component affecting server performance. Having all database tables in memory is the ideal scenario. This is not possible obviously with databases of any significant size, so the throughput of the disk subsystem is very important.<br />
<br />
Disk hardware configuration Starting with SQL Server 2000 and continuing with SQL 2005, when SQL Server does read-ahead for table or index scans, it will perform up to 1 MB reads. This can adversely affect performance on arrays configured with a small stripe size. For example, an array with an 8 KB stripe size would do 128 disk reads for each 1 MB read-ahead request. Because SQL Server can queue 128 to 256 requests, an improperly disk array can cause an extremely large number unnecessary disk requests. SQL Server only issues larger then 512 KB I/O reads when Windows DataCenter is used. For all non-DataCenter Environments, a stripe size of 512 KB is recommended for the disk subsystem with SQL Server. I/O bottlenecks can also exacerbate database locks and contention, so a sound disk subsystem will also reduce help to reduce locking and improve database responsiveness to users. For the best I/O performance, a greater number of smaller, faster disks will always yield better results than fewer, larger, slower disks. This especially applies when configuring multiple disks into an array, regardless of RAID configuration. Always use disks of the same size and speed in a given array. Storage area network (SAN) environments are becoming increasingly popular for large, mission-critical SQL Server databases. With their advanced caching technology and enhanced manageability, SAN storage based on Fibre Channel disks typically offers very high performance and availability — albeit it at considerably more cost than direct attached storage (DAS). SAN is often the optimal solution where peak performance, scalability and system availability are critical requirements. Particularly critical is the configuration of the system page file. This is discussed in detail in 11.5, “Virtual memory” on page 305.<br />
<br />
Disk partitioning Operating system disks should where possible be kept on separate physical disks that those of the SQL Server databases and log files. A pair of lower capacity RAID-1 (mirrored) disks is typically best for the operating system files.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
881<br />
<br />
SQL Server log files should always be physically separated onto different arrays from all other database files. For SQL Server systems with very multiple busy databases, transaction log files should be physically separated from each other also. Transaction logging is primarily sequential write I/O, favoring RAID-1 or RAID-1E. RAID-5 and RAID-5E are not recommended. Given the criticality of the log files RAID-0 is not recommended, despite its improved performance. There is considerable I/O performance benefits to be gained from separating transaction logging activity from other random disk I/O activity. Doing so allows the hard drives including the log files to concentrate on sequential I/O. Note that there are times when the transaction log will need to be read as part of SQL Server operations such as replication, rollbacks, and deferred updates. SQL Servers that participate in replication should pay particular attention to making sure that all transaction log files have sufficient disk I/O processing power because of the read operations that frequently occur. There is additional administration involved with physically separating SQL Server objects from the rest of their associated database through SQL Server files and file groups. For the purposes of investigating very active tables and indexes, this might be very worthwhile. By separating tables or indexes away from all other database objects, accurate assessments can be made of the I/O requirements of that object. This is not as easy to do when all database objects are placed within one large drive pool. This type of physical I/O separation might be appropriate during database development and benchmarking so that database I/O information can be gathered and applied to capacity planning for the production database server environment. The following areas of SQL Server activity can be separated across different hard drives, arrays, disk controllers and disk channels (or combinations thereof where possible): <br />
<br />
Transaction log files Tempdb Database files Tables associated with a lot of query or write activity Non-clustered indexes associated with a lot of query or write activity<br />
<br />
The physical separation of SQL Server I/O activities is quite convenient with the use of hardware RAID controllers such as the IBM ServeRAID and Fibre Channel (DS4000 series) options. The approach that provides the most flexibility is arranging the RAID controllers so that a separate RAID SCSI channel is provided for each of the separate SQL activities mentioned above. Windows logical volumes are associated to each RAID array, and SQL Server files can be separated between distinct RAID arrays based on known I/O usage patterns.<br />
<br />
882<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
With this configuration, it is possible to associate disk queuing with a distinct disk channel, because System Monitor reports the queuing behavior during load testing or heavy production loads. Because ServeRAID and DS4000 controllers and EXP disk expansion enclosures support online RAID expansion, disk queuing on such RAID arrays can potentially resolved by adding more drives to a given array until System Monitor reports that disk queuing for that array has reached acceptable levels. This process can be even be carried out while SQL Server is online, however it will impact performance for the duration of the expansion process. Tempdb is a default database created by SQL Server to be used as a shared working area for a variety of activities, including temporary tables, sorting, subqueries, and aggregates with GROUP BY or ORDER BY queries using DISTINCT (temporary worktables have to be created to remove duplicate rows), cursors, and hash joins. It is good to enable tempdb I/O operations to occur in parallel to the I/O operations of related transactions. As tempdb is a scratch area and very update intensive, use RAID-0, RAID-1 or RAID-1E, or RAID-4DP to achieve optimal performance benefits. RAID-5 or RAID-5E are not recommended. As tempdb is reconstructed with each server restart, RAID-0 is a sound option on production SQL Servers as redundancy is secondary to performance. The only concern with using RAID-0 for tempdb in a production environment is that SQL Server would need to be restarted if any physical drive failure were to occur in the RAID-0 array. If very high availability is a requirement than RAID-1 or RAID-1E are better choices, at the expense of slight reduced performance. The ALTER DATABASE command can be used to change the physical file location of the SQL Server logical file name associated with tempdb and, thus, the actual tempdb database. The master, msdb, and model databases are not used heavily during production compared to user databases. It is typically not necessary to consider them in I/O performance tuning considerations. For example, the master database is normally used just for adding new logins, databases, devices, and other system objects. Accordingly, performance for these databases is not critical. Non-clustered indexes reside in B-tree structures, which can be separated from their related database tables also with the ALTER DATABASE command.<br />
<br />
Detecting SQL 2005 bottlenecks If CPU utilization is low, but response time or run time is not okay, you might have a disk bottleneck. The Physical Disk Perfmon counters might provide the insight needed for tuning most of the sequential I/O workloads. However, in the OLTP workload, the disk load of the log and checkpoint processes are adjusted by SQL<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
883<br />
<br />
Server and so are invisible to Perfmon. When writing to the log, the disk queue length is kept at 1. Likewise, the checkpoint is controlled so that it does not swamp the disk and severely reduce throughput. In this case, use the query in Example 26-1 to capture reads, writes, and I/O stalls by database file which is a feature that is only available with SQL Server 2005. Example 26-1 SQL Server I/O statistics by file<br />
<br />
select m.name, v.*, m.physical_name h from sys.dm_io_virtual_file_stats (null, null) v ,sys.master_files m xwhere v.database_id = m.database_id gggand v.file_id = m.file_id Using this information, you can compute reads per second, writes per second, read and write block sizes, and average milliseconds of wait time (I/O stalls) for read and write. If these wait times are large, putting the files in faster disk arrays can improve database performance. Using one of the new dynamic management views (DMV) in SQL Server 2005, it is now possible to obtain I/O statistics by individual table and index, which can be used to make decisions about dividing up the tables into file groups and for application level tuning (Example 26-2). Example 26-2 SQL Server I/O statistics by table/index<br />
<br />
select o.name, i. ggfrom sys.dm_db_index_operational_stats(db_id, null, null, null) i ,sys.objects o gwhere i.object_id = o.object_id hhhand i.object_id > 100 Refer to 19.4, “Disk bottlenecks” on page 667 for more information about detecting and removing disk subsystem-related bottlenecks.<br />
<br />
26.6.3 Processor SQL Server requires a lot of processor cycles to perform many database activities. SQL Server takes advantage of the SMP capabilities of Windows on servers with multiple processors by implementing tasks with multiple threads. If the total processing percentage (% Processor Time) is higher than 80% for sustained periods, then you might have a CPU bottleneck. Normally, the kernel mode time (% Privileged Time) is low for SQL Server. If it is high, there might be a problem with a disk driver or the disk subsystem.<br />
<br />
884<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To see if SQL Server has work that is queued that could run but is waiting for a busy CPU, you must check inside SQL Server because SQL Server does not queue its work to the Windows operating system. SQL Server has a user-mode scheduler in which it queues tasks that are waiting. This query results in an instantaneous view of the number of tasks queued in SQL Server 2005 that can run (Example 26-3). Example 26-3 Tasks that can be run in SQL Server 2005<br />
<br />
select sum(runnable_tasks_count) from sys.dm_os_schedulers where scheduler_id < 255 If SQL Server frequently has a non-zero number of runnable tasks, then adding additional processors likely results in better throughput. To improve processing power on a server, you can: Add more CPUs Replace existing CPUs with processors of faster clock-speed Apply affinitization methods to more effectively use processing power Servers with a high CPU utilization and low queue length will likely benefit from faster CPUs. High CPU utilization combined with high queue lengths perform better by adding more processors. A longer queue length means there is more work to be done by the CPUs, and this is best served my adding more processors. A faster clock speed alone means the work will get done more quickly; however, there will still be more work than fewer processors can efficiently handle. Adding more CPUs generally increases SQL Server performance as the server is able to do more at the same time effectively. Databases such as SQL Server make use of Level 2 and Level 3 cache on modern servers very efficiently. Larger caches means shorter wait times reading and writing data to and from main system memory. For highest performance, implement CPUs with as large L2 caches as budget permits. Servers with more processors are able to take advantage of larger L2 caches than servers with fewer processors<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
885<br />
<br />
As with memory configuration settings, SQL Server is best left to auto-tune and manage resource usage of systems processors using default settings. Should you need to be more specific in how SQL Server uses system processor resources, you should follow these steps carefully. 1. Access the processor configuration panel for SQL Server 2000. Open SQL Server Enterprise Manager. 2. In the left frame, expand Microsoft SQL Servers, and locate the SQL Server Group for the server that you want to configure. 3. Right-click the SQL Server in the right-frame and click Properties. 4. Select the Processor tab (Figure 26-7).<br />
<br />
The default settings of all enabling installed processors for use by SQL Server and parallelism are best for most servers.<br />
<br />
Figure 26-7 SQL Server 2000 processor usage configuration<br />
<br />
This tab is divided into two sections, Processor control and Parallelism, which we discuss in the following sections.<br />
<br />
886<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Processor control You select the actual physical processors on which SQL Server executes by selecting the physical processor from the list, as shown in Figure 26-7. The default setting has all processors selected. There are only two likely instances in which you might choose to not use all processors: You have an instance of another application or process running on the server that is bound to a particular processor and you want to free up this processor to focus on servicing this task. You are using per-processor licensing and do not have a licence for all processors in a given physical server. Under other conditions, however, adjusting the SQL Server processor affinity in this manner will more than likely hinder rather than help performance. The Maximum worker threads setting specifies the maximum number of worker threads that are available to SQL Server processes. The default setting is 255 and is expected to increase particularly as the number of cores per processor increases. Note that the default setting of 255 does not restrict the number of user connections to 255. This setting means that only a maximum of 255 user queries can run at any given time. However, as the number of queries need to be serviced by a much smaller number of physical processors, the number actually processing concurrently is considerably less anyway. The pool of worker threads really only needs to be large enough to service the number of user connections that are actively executing at the same time. Setting the value too high means additional resources are being allocated unnecessarily to threads that are not being serviced. By running the sp_who command in query analyzer, you can determine the number of SQL threads that are currently active. If all of them are active, it might make sense to increase the number of maximum worker threads. The Boost SQL Server priority on Windows setting changes the processor priority for the SQL Server threads from Normal to High. The Windows operating system schedules threads for execution based on a numeric priority ranging from 1 through 31 (0 is reserved for operating system use). When several threads are waiting to execute, the thread with the highest priority is serviced first. When the option is not selected, the processor priority is Normal and has a value of 7. In almost all instances, leaving SQL Server to run at a Normal processor priority produces good performance. It also gives SQL Server threads a high enough priority to get CPU resources without adversely affecting other applications.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
887<br />
<br />
When the option is selected, the priority of SQL Server threads is set to High, which has a value of 13. This setting gives SQL Server threads a higher priority than most other applications on the server. Thus, SQL Server threads will tend to be dispatched whenever they are ready to run and will not be preempted by threads from other applications. The boost option can improve performance when a server is only running SQL Server and no other applications. If a memory-intensive operation occurs in SQL Server, however, other applications are not likely to have a high-enough priority to preempt the SQL Server thread and might wait some time to be serviced.<br />
<br />
Use Windows NT fibers is a configuration setting also known as light-weight pooling or fiber mode. Under normal conditions, SQL Server uses only one thread per active user process, which is sometimes called thread mode. SQL Server in this instance uses User Model Schedulers (UMS) threads to run user processes. The UMS attempts to balance the number of threads per user process, and the number of user processes across the CPUs in the server. Selecting Use Windows NT fibers allows SQL Server to execute several concurrent tasks (or fibers) within the same thread. Switching between actual threads requires context switches between the user mode of the application code and the kernel mode of the thread manager. This is a moderately expensive operation on the CPU. Fibers are a subcomponent of threads in Windows that are managed by code running in user mode. Switching fibers does not require the user-mode to switch to the kernel-mode transition needed to switch threads. That is, it is all handled within the application user mode The scheduling of fibers is managed by the application, while the scheduling of threads is managed by Windows. Normally, without fiber mode enabled, SQL Server schedules a thread per concurrent user request up to a maximum of maximum worker threads, specified above. With fiber mode enabled, SQL Server allocates one thread per CPU then allocates a fiber per concurrent user request, up to maximum worker threads. This adds a potential performance boost to an already busy server. Enabling Windows NT fibers will really only make a difference on large multi-processor servers that are running at or near maximum capacity and are experiencing considerable context-switching (greater than 20,000 per second). In some instances, enabling Windows NT fibers can actually degrade performance. Threads that cannot take advantage of fiber-mode cause the server to switch between thread-mode and fiber-mode, negatively impacting the SQL Server performance. Thus, you should monitor the effect of enabling Windows NT fibers before and after the change.<br />
<br />
888<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Parallelism The parallelism settings determine how many processors should be used in the parallel execution of user queries. In an SMP environment, SQL Server can take a single query—if it meets given conditions—and split it between multiple CPUs, enhancing performance considerably. In almost all instances, the best option to select it the default setting of Use all<br />
<br />
available processors. A CPU-bound query that must examine a large number of rows often benefits if portions of its execution plan are run in parallel. SQL Server determines automatically which queries will benefit from parallelism and generates a parallel execution plan. If multiple processors are available when the query begins executing, the work is divided across the processors. If you need to adjust the number of processors on which parallel queries can execute, select Use X processors and choose the number of processors desired. The number of processors that are available in this selection is determined by the number of processors selected for use with SQL Server in the Processor Control frame. A parallel query execution plan can use more than one thread, while a serial execution plan, used by a nonparallel query, only uses a single thread for its execution. The actual number of threads used by a parallel query is determined at query plan execution and initialization and is called the degree of parallelism. The Minimum query plan threshold for considering queries for parallel execution (cost estimate) setting specifies the threshold at which SQL Server creates and executes parallel plans. SQL Server creates and executes a parallel plan for a query only when the estimated cost to execute a serial plan for the same query is higher than the value set for this option. The default setting of 5 is suitable for almost all environments and should only be changed after consideration and monitoring of your SQL Server environment.<br />
<br />
Processor I/O affinity for SQL Server 2005 By default, no specific processor affinity is set in SQL Server 2005, and all processors can be scheduled to perform all tasks. If you want to precisely define how system resources are used, you can enable hard processor affinity in SQL Server 2005. You can configure the affinity setting per database instance. There are two ways to define an affinity mask in SQL Server 2005: Use SQL Server Management Studio graphical interface. Use the sp_configure stored procedure.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
889<br />
<br />
To change processor affinity, do the following: 1. Select Start → Programs → Microsoft SQL Server 2005 → SQL Server Management Studio. The Server Properties window opens. 2. Connect to the SQL Server instance. 3. Right-click the instance icon in Object Explore and click Properties. The window that is shown in Figure 26-8 opens. 4. Click Processors in Select a Page in the left pane. 5. Select or clear the check marks in the Processor Affinity column to specify which processors that you want this particular database instance to use.<br />
<br />
Figure 26-8 Processor affinity and I/O affinity in SQL Server 2005<br />
<br />
890<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
You can also set affinity to disk I/O by using I/O affinity. I/O affinity can associate SQL Server disk I/O to a specified subset of processors so that specified processors handle all disk I/O. This option can work effectively in the OLTP environment where high load is generated, especially in cases of a server with 16 or more processors. I/O affinity can enhance the performance of SQL Server threads that are issuing I/O. For example, you could assign CPU 1 for I/O affinity and all the other CPUs for processor affinity. However, I/O affinity does not always improve performance, so you must be careful before using it. Note: This function does not support hardware affinity for individual disks or disk controllers. We can configure I/O affinity in either SQL Server Management Studio (as shown in Figure 26-8 on page 890) or the sp_configure stored procedure. Refer to 19.2, “CPU bottlenecks” on page 656 for more information about detecting and removing processor-related bottlenecks.<br />
<br />
26.6.4 Network The network subsystem is usually the least taxed in SQL Server database environment, but due care should still be taken to implement the best network configuration possible to ensure it does not unknowingly become a bottleneck. Where possible, connect your SQL Server to a switch running at 1000 MBps and full-duplex. Multiple network interfaces can also offer redundancy and network throughput (aggregation) enhancements. Of particular importance is the throughput between application front ends and Web servers that need to communicate quickly with backend SQL Server databases. In these instances, ensure high-speed network interfaces and switching technology are employed. Limit the number of protocols for the server network interfaces and try to keep the number of router “hops” between clients and servers to a minimum if you can. With SQL Server 2005, a Perfmon SQL Server counter object called Wait Statistics includes a counter called Network IO Waits, which might be useful to monitor the network traffic.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
891<br />
<br />
SQL Server 2005 Network affinity Network affinity is a new feature in SQL Server 2005. This feature provides clients with the ability to connect to specific nodes. Tip: In the SQL Server Books Online, this network affinity is called NUMA affinity. For example, consider a 2-node, 8-processor system and two network cards in the server, one in each node. In this case, we can configure the IP address that is associated with network card 1 to use the processors on node 1 and the IP address that is associated with network card 2 to use the processors on node 2. We can configure these settings in the SQL Server Configuration Manager GUI or the Windows registry. When this setting is used, the workload of IP address 1 can only be processed by processors 0 to 3 in node 1. In the same way, the workload of IP address 2 is only processed by processors 4 to 7 in node 2. Thus, each workload takes full advantage of local memory access because the requested data is located in local memory whenever possible. If the memory in the local node is insufficient, then memory in another node is allocated. To configure network affinity in SQL Server 2005, use a binary mask that represents nodes. The mask has a bit for each node as 76543210, with the first node as zero (Figure 26-9). Set each node bit to 1 to select a node or 0 to not select a node. For example: To specify just node 0, use mask 00000001 or hex 0x1. To specify just node 1, use mask 00000010 or hex 0x2. To specify nodes 0, 2, and 4, use mask 00010101 or hex 0x15. To specify all nodes, use a mask of -1 or leave the field blank.<br />
<br />
Node 7<br />
<br />
Node 0<br />
<br />
76543210 -------00000010<br />
<br />
Specifies node 1 only In hex: 0x2<br />
<br />
Figure 26-9 Determining the affinity mask to use for network affinity<br />
<br />
892<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Tip: You do not have to use different port numbers. In our example, we configured the affinity of a 2-node (8-way) by mapping node 1 to 9.42.171.184, port 1444 and node 2 to 9.42.171.160, port 1500. You can do this as follows: 1. Select Start → Programs → Microsoft SQL Server 2005 → Configuration Tools → SQL Server Configuration Manager. 2. In SQL Server Configuration Manager, expand SQL Server 2005 Network Configuration and expand Protocols for <instance name>, where <instance name> is the database instance that you want to configure. 3. In the details pane, right-click the IP address that you want to configure, and click Properties. 4. To associate a combination of IP addresses and port numbers to specific nodes, set Listen All to No in the Protocol tab, as shown in Figure 26-10. The default is Yes.<br />
<br />
Figure 26-10 Protocol setting for affinity in SQL Server Configuration Manager<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
893<br />
<br />
5. Click the IP Addresses tab (Figure 26-11). For each network adapter, enter the port number and node affinity mask in the form: port[mask] So, for network adapter 1, we want to instruct SQL Server to listen on port 1444 and have the processors in node 1 handle all requests. The value in the TCP Port field is: 1444[0x1]<br />
<br />
Figure 26-11 IP Address setting for NUMA affinity in SQL Server Configuration Manager<br />
<br />
Note: If you leave Listen All as Yes (the default) in Figure 26-10 on page 893, the values in the corresponding fields in IPALL in the IP Addresses tab is applied to all IP addresses that are listed in the IP Addresses tab, as shown in Figure 26-11. 6. Restart SQL Server 2005 service so that the changes take affect.<br />
<br />
894<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
26.6.5 Hardware tuning versus application and database design While hardware optimization is critical in tuning SQL Server performance, remember always that proper application and database design and configuration usually form a more significant component in overall database performance. SQL Server performance will not always be improved by simply spending more money on improving hardware. Be sure that this is the bottleneck of any server before committing financial resources.<br />
<br />
26.7 Scaling SQL Server SQL Server 2000 teamed with Windows Server 2003 offers high-end hardware support options in terms of memory and CPU scaling. So, is it better to scale a SQL Server up or out?<br />
<br />
Scale-out refers to the idea of increasing processing capacity by adding additional servers to a solution. Adding servers to a Web farm to handle larger numbers of users is a good example of scale-out. A user can connect to any Web server, because there is no shared data and the Web applications are stateless. Scale-out is not typically a database solution; however, SQL Server can do scale-out using distributed partitioned views, also called federated databases. Scale-out can work well when the data can be partitioned on a natural boundary, such as a geographical region or division of a company. However, managing a distributed partition configuration can require significant manual effort.<br />
<br />
Scale-up refers to the idea of increasing processing capacity by adding additional processors (and memory and I/O bandwidth) to a single server, making it more powerful. Scaling-up is precisely what the x3950 is designed to do: to scale up by adding chassis to a hardware partition to form 2-node, 4-node, and 8-node configurations. SQL Server 2005 is also designed for scale-up. (For more information about large scale-up systems and SQL Server 2005, see the IBM Redpaper, SQL Server 2005 on the IBM Eserver xSeries 460 Enterprise Server, REDP-4093.) Thus, when your database application needs to scale-up to handle increased demands, SQL Server 2005 can grow from a 1-node or 2-node server to an enterprise-class 8-node server with 32 processor sockets and 512 GB of physical memory, of which the x64 SQL Server 2005 is designed to take full advantage. Many of the performance gains in very high-end enterprise systems might only be truly realized by using the 64-bit (AMD64, EM64T, x64, or Itanium) editions of<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
895<br />
<br />
SQL Server and Windows Server 2003. With its support for extra large direct memory access in particular, only when using this version of SQL Server will the true performance benefits of extra-large hardware configurations in fewer, bigger servers be yielded. If massive performance benefits are required 64-bit extension technology can be the optimal solution.<br />
<br />
26.8 Optimizing Windows for SQL Server You can optimize the performance of SQL Server by changing various operating system settings within Windows. We recommend that you implement these changes after you have installed SQL Server. The operating system should be configured to service the unique demands of a database application server. You can find more operating system tuning specifics in Chapter 11, “Microsoft Windows Server” on page 295. We discuss each of the parameters that we refer to here in more detail in that chapter.<br />
<br />
26.8.1 Processor scheduling A control panel option exists to determine how Windows allocates processor time between foreground programs and background processes. To change this setting in Windows Server 2003, do the following: 1. Open the System Control Panel. 2. Select the Advanced tab. 3. Within the Performance frame, click Settings. 4. Select the Advanced tab (Figure 26-12).<br />
<br />
896<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Set the application response to Background services to improve background application performance<br />
<br />
Figure 26-12 Configuring processor scheduling to favor background processes<br />
<br />
26.8.2 System cache You can control how Windows manages the size and priority of the file system cache. SQL Server does not take advantage of the Windows file system cache because it creates its own memory-resident cache. Thus, it is best to minimize the size of the file system cache using the following settings: 1. Click Start → Control Panel → Network Connections. 2. While still in the Start menu context, right-click Network Connections and choose Open. 3. Select any of Local Area Connections. This setting affects all LAN interfaces, so which LAN connection you choose in the above steps is not important. 4. Right-click the selected connection object and choose Properties. 5. Select File and Printer Sharing for Microsoft Networks. 6. Click Properties. 7. Select Maximize Data throughput for network applications.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
897<br />
<br />
This allocates memory for network connections to your server and less is set aside for cache.<br />
<br />
Set this option for SQL Server<br />
<br />
Figure 26-13 Setting Windows 2000 to an application server role<br />
<br />
These settings help in overall SQL Server performance as it manages its own memory environment. By reducing memory allocation to the file system cache, SQL Server is able to make use of more system memory, and the memory assigned to the working set of applications is treated with a greater priority than that of the file system cache. Note: If your server has a large amount of memory, then you might not realize a performance advantage by selecting Maximize data throughput for network applications over Maximize data throughput for file sharing. See 11.6.1, “Servers with large amounts of free physical memory” on page 314 for more information.<br />
<br />
26.8.3 Virtual memory Windows employs a virtual memory management system that allows disk space to be used when physical memory is constrained. Virtual memory allows applications to address more memory than is physically installed in the server. A pagefile runs at its most optimal when distributed on multiple, dedicated, separate physical volumes. For best performance, you should configure page files with the same initial and maximum size to avoid the system intensive process of the operating system resizing the page file dynamically.<br />
<br />
898<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To configure the page file size: 1. Open the System Control Panel. 2. Select the Advanced tab. 3. Within the Performance frame, click Settings. 4. Select the Advanced tab. 5. Click Change. The window shown in Figure 26-14 opens.<br />
<br />
Create paging files on multiple drives to improve paging performance.<br />
<br />
Figure 26-14 Configuring the page file settings in Windows<br />
<br />
For best performance, avoid placing a page file on the same disk drive together with I/O-intensive SQL Server files such as database files or online redo logs. For further information about optimizing page file performance, see 11.5, “Virtual memory” on page 305.<br />
<br />
26.9 Further SQL Server optimizations With every new release of SQL Server, the number of performance settings that can be manually configured reduces, and SQL Server becomes more proficient<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
899<br />
<br />
at self-tuning itself. As noted elsewhere, good application and database design will more often produce better performance than configuration tuning will. Nonetheless, SQL Server includes a number of configuration panels that can be manually tuned to optimize system performance. We discuss these control panels and the purpose of their most important settings in this section. Note: You can make many of these changes that have a GUI dialog in SQL Server using the sp_configure stored procedure. Many of these changes require you to restart SQL Server, meaning that you need to shutdown all databases. In some instances, you might also need to reboot the server. For more information about sp_configure, see “Stored Procedures - System Stored Procedures” in the Transact-SQL Reference in SQL Server Books Online. A program shortcut is available as part of the SQL Server program group in the Windows Start menu.<br />
<br />
26.9.1 The max async I/O option (SQL Server 7.0 only) The max async I/O option specifies the number of simultaneous disk transfers that are issued by SQL Server to the disk subsystem. The default is 32. For high-end RAID controllers, this default value can restrict disk subsystem performance. Note: This parameter is only available in SQL Server 7.0; it has been removed starting SQL Server 2000. A rule of thumb for setting max async I/O for SQL Servers that are running on larger disk subsystems is: Max async I/O = [number of physical disk drives used by SQL Server] x 3 The recommended multiplier is 2 or 3. After changing this value, watch Performance console for signs of disk activity or queuing issues. A good value for max async I/O is one that allows a checkpoint to be fast enough. The goal is to make a checkpoint fast enough to finish before another checkpoint is needed (based upon desired recovery characteristics), but not so fast that the disks cannot keep up and that command queuing occurs. The negative impact of setting this configuration option too high is that it can cause a checkpoint to monopolize disk subsystem bandwidth that is required by other SQL Server I/O operations, such as reads.<br />
<br />
900<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To set this value, execute the following command in SQL Server Query Analyzer: sp_configure 'max async io', [value] In this command, [value] is expressed as the number of simultaneous disk I/O requests that the SQL Server system will be able to submit to Windows during a checkpoint operation, which in turn submits the requests to the physical disk subsystem. This configuration option is dynamic. It does not require a stop and restart of SQL Server to take effect.<br />
<br />
26.9.2 LazyWriter SQL Server LazyWriter helps to produce free buffers, which are 8 KB data cache pages void of data. As LazyWriter flushes each 8 KB cache buffer out to disk, it needs to initialize the cache page's identity so that other data can be written into the free buffer. LazyWriter aims to produce free buffers during periods of low disk I/O, such that disk I/O resources are readily available for use and there will be minimal impact on other SQL Server operations. SQL Server configures and manages the level of free buffers automatically. Monitor the System Monitor object, shown in Table 26-1, to see if this value drops. Table 26-1 Counter for free buffers<br />
<br />
Counter<br />
<br />
Should be<br />
<br />
SQL Server: Buffer Manager-Free Buffers<br />
<br />
Consistent, non-zero at all times<br />
<br />
Optimally, LazyWriter keeps this counter steady throughout SQL Server operations. This indicates that the LazyWriter is keeping up with the system demand for free buffers. It is not desirable for this counter to reach zero, because this indicates there were times when the system load demanded a higher level of free buffers than the SQL Server's LazyWriter was able to provide. If LazyWriter is having problems keeping the free buffer steady, or at least above zero, it could mean that the disk subsystem is not able to provide LazyWriter with the disk I/O performance needed. Compare drops in SQL Server free buffer level to any disk queuing at the hardware level to see if this is true. The optimal solution is to add more physical disk drives to the database server disk subsystem in order to provide more disk I/O processing power.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
901<br />
<br />
Monitor the current level of disk queuing in the Windows System Console by examining the counters listed in Table 26-2. Table 26-2 Counters for disk queuing<br />
<br />
Counter<br />
<br />
Should be<br />
<br />
PhysicalDisk: Average Disk Queue Length<br />
<br />
Less than 2. If the disk subsystem is RAID-based, then 2xN where N is the number of physical drives in the array.<br />
<br />
PhysicalDisk: Current Disk Queue Length<br />
<br />
Less than 2. If the disk subsystem is RAID-based, then 2xN where N is the number of physical drives in the array.<br />
<br />
Ensure that the disk queue length is less than two for each physical drive associated with any SQL Server activity, or 2xN for RAID arrays, where N is the number of drives in array. You can adjust LazyWriter disk I/O request behavior in SQL Server by using the max async I/O option of the sp_configure command, which controls the number of 8 KB disk write requests (including requests coming in from LazyWriter, checkpoint, and the worker threads) that SQL Server can simultaneously submit to Windows and, in turn, to the disk I/O subsystem. If disk queuing occurs at unacceptable levels, decrease max async I/O. If it is imperative that SQL Server maintain its currently configured level of max async I/O, either add faster disks or more physical disks to the disk subsystem until disk queuing comes down to acceptable levels. The max async I/O parameter is self-tuning in SQL Server 2000 and 2005.<br />
<br />
26.9.3 Checkpoint Checkpoint writes dirty pages out to the SQL Server data files. Dirty pages are any buffer cache pages that have been modified since being brought into the buffer cache. A buffer written to disk by checkpoint still includes the page and users can read or update it without rereading it from disk, which is not the case for free buffers created by LazyWriter. Checkpoint aims to let worker threads and LazyWriter do the majority of the work writing out dirty pages. Checkpoint does this by trying an extra checkpoint wait before writing out a dirty page if possible. This extra checkpoint wait provides the worker threads and LazyWriter more time in which to write out the dirty pages. The conditions under which this extra wait time for a dirty page occurs is detailed in SQL Server Books Online under the section Checkpoints and the Active Portion of the Log. The main idea to remember is that checkpoint aims to even<br />
<br />
902<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
out SQL Server disk I/O activity over a longer time period with this extra checkpoint wait. To make checkpoint more efficient when there are a large number of pages to flush out of cache, SQL Server sorts the data pages to be flushed in the order that the pages appear on disk. Flushing the page in order helps to minimize disk arm movement during cache flush and potentially takes advantage of sequential disk I/O. Checkpoint also submits 8 KB disk I/O requests asynchronously to the disk subsystem. This allows SQL Server to finish submitting required disk I/O requests faster because checkpoint does not wait for the disk subsystem to report back that the data has been actually written to disk. It is important to monitor disk queuing on all hard drives associated with SQL Server data files to notice if SQL Server is sending down more disk I/O requests than the disk subsystem can handle. If this condition is occurring, more disk I/O capacity must be added to the disk subsystem so that it can handle the load. Monitor the current level of disk queuing in Performance console by looking at the counters listed in Table 26-3. Table 26-3 Counters for Checkpoint<br />
<br />
Counter<br />
<br />
Should be<br />
<br />
PhysicalDisk: Average Disk Queue Length<br />
<br />
Less than 2. If the disk subsystem is RAID-based, then 2 x n where n is the number of drives in the array.<br />
<br />
PhysicalDisk: Current Disk Queue Length<br />
<br />
Less than 2. If the disk subsystem is RAID-based, then 2 x n where n is the number of drives in the array.<br />
<br />
SQL Server allows adjustment of checkpoint's dirty page flushing behavior with the use of the max async I/O option of the sp_configure command, which controls the number of 8 KB cache flushes that checkpoint can submit simultaneously to Windows (and, in turn, to the disk I/O subsystem). If disk queuing occurs at unacceptable levels, decrease max async I/O. If it is imperative that SQL Server maintains its currently configured level of max async I/O, add faster and /or more disks to the disk subsystem until disk queuing comes down to acceptable levels. Conversely, if it is necessary to increase the speed with which SQL Server executes checkpoint and the disk subsystem is already powerful enough to handle the increased disk I/O and continue to avoid disk queuing, increase max async I/O to allow SQL Server to send more disk I/O requests at the same time, potentially improving I/O performance.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
903<br />
<br />
Watch the disk queuing counters carefully after modifying the value of max async I/O. In particular, monitor disk read queuing and disk write queuing. If max async I/O is set too high for a given disk subsystem, checkpoint might tend to queue a large number of disk write I/O requests. This can cause SQL Server read activity to be blocked and impact performance. The max async I/O parameter is self-tuning in SQL Server 2000 and 2005.<br />
<br />
26.9.4 Log manager Like all other major relational databases, SQL Server ensures that all write activities (insert, update, and delete) performed on the database will not be lost if something interrupts SQL Server's online status, such as a power failure or disk drive failure. One feature of SQL Server that helps guarantee recoverability is the built-in logging process. Before any implicit (single SQL query) or explicit (defined transaction that issues a BEGIN TRAN/COMMIT, or ROLLBACK command sequence) transaction can be completed, SQL Server's log manager must receive a signal from the disk subsystem that all data changes associated with that transaction have been written successfully to the associated log file. This rule guarantees that if the SQL Server is abruptly shut down and the transactions written into the data cache are not yet flushed to the data files (remember that flushing data buffers are a function of Checkpoint’s or LazyWriter), the transaction log can be read and reapplied to the database when the system is restored. Reading the transaction log and applying the transactions to SQL Server after an unexpected server stoppage is referred to as a recovery process. Because SQL Server must wait for the disk subsystem to complete I/O to the log files as each transaction is completed, it is important that the disks including the log files have sufficient disk I/O handling capacity for the anticipated transaction load. Obviously, a busy database will generate more transactions and thus warrants a disk subsystem able to cope with the load this imposes. The method for monitoring disk queuing that is associated with log files differs from that used for database files. Use the following System Monitor console counters to see if there are log writer requests waiting on the disk subsystem for completion: SQL Server: Databases <database instance>: Log Flush Waits Times SQL Server: Databases <database instance>: Log Flush Waits/sec To optimize performance, the System x ServeRAID controller controlling the disks hosting the log files should be configured for write-back mode. To ensure data is protected it is recommended to make use of a disk controller with a battery-backed up cache installed and enabled.<br />
<br />
904<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
26.9.5 Read-ahead manager SQL Server’s read-ahead manager is completely self-configuring and self-tuning. Read-ahead manager is tightly integrated with the operations of the SQL Server query processor. The query processor communicates situations that would benefit from read-ahead scans the to read-ahead manager. Large table scans, large index range scans, and probes into clustered and non-clustered index B-trees are situations that would benefit from a read-ahead. This is because read-ahead occur with 64 KB I/Os, which provide higher disk throughput potential for the disk subsystem than do 8 KB I/Os. When it is necessary to retrieve a large amount of data from SQL Server, read-ahead is the best way method for doing so. Read-ahead manager benefits from the simpler and more efficient index allocation map (IAM) storage structure. The IAM is SQL Server's new method of recording the location of extents (eight pages of SQL Server data or index information for a total of 64 KB of information per extent). The IAM is an 8 KB page that tightly packs information (through a bitmap) about which extents within the range of extents covered by the IAM include required data. The compact nature of IAM pages makes them fast to read and tends to keep regularly used IAM pages in buffer cache. Combining the query information from the query processor and quickly retrieving the location of all extents that need to be read from the IAM pages, the read-ahead manager can construct multiple sequential read requests. Sequential 64 KB disk reads provide extremely good disk I/O performance. The important System Monitor counter to watch with respect to SQL Server read-ahead manager is: SQL Server: Buffer Manager-Read-Ahead Pages More information can be found by executing the command DBCC PERFMON (IOSTATS): RA Pages Found in Cache RA Pages Placed in Cache If the page is already hashed (that is, the application read it in first and read-ahead wasted a read), it goes against the page found in cache count. If the page is not already hashed (that is, a successful read-ahead), it counts towards pages placed in cache. One caveat with the read-ahead manager is that too much read-ahead can be detrimental overall to performance because it can fill cache with pages that were not needed, loading the processor and I/O subsystems unnecessarily. The<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
905<br />
<br />
solution is a general performance tuning goal—ensure that all SQL queries are tuned such that a minimal number of pages are brought into buffer cache. This tuning includes making sure that you use the right index for the right task. Save clustered indexes for efficient range scans and define non-clustered indexes to help quickly locate single rows or smaller rowsets. Note that if you only plan to have one index in a table and that index is for the purposes of fetching single rows or smaller rowsets, you should make the index clustered because clustered indexes will be faster than non-clustered indexes (but not by the same dramatic scale as is the case for range scans). As described in Chapter 9, “Disk subsystem” on page 169, disk I/O requests are typically for data that is not sequential on the disk because servers typically satisfy requests from multiple users at the same time. The SQL Server read-ahead manager attempts to perform disk I/O in such a way as to arrange its reads such that read-ahead scans are done sequentially (often referred to as “serially” or “in disk order”). While read-ahead manager aims to perform I/O operations sequentially, page splitting will tend to cause extents to be read randomly versus sequentially. This is one reason why it is important to eliminate and prevent page splitting. Read-ahead manager typically performs I/O in 64 KB chunks. Log manager will write sequentially to the log files in sizes ranging up to 32 KB.<br />
<br />
26.9.6 Address Windowing Extensions support 32-bit CPUs can only address up to 4 GB of RAM because of the limited address space of these processors. To get around this limitation, SQL Server 2000 and SQL Server 2005, Enterprise Edition supports a feature called Address Windowing Extensions (AWE) that allows up to 128 GB of RAM to be addressed. In addition to enabling support for AWE in the hardware and operating system (see 11.14, “Using PAE and AWE to access memory above 4 GB (32-bit x86)” on page 335), you need to enable AWE support in SQL Server. AWE support is not enabled automatically. To turn AWE support on, you must set the awe enabled advanced SQL Server 2000 or 2005 option to 1.<br />
<br />
906<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
To turn on AWE support from within SQL Server, issue the commands that are shown in Example 26-4. Example 26-4 Command sequence to enable AWE support<br />
<br />
SP_CONFIGURE 'show advanced options', 1 RECONFIGURE GO SP_CONFIGURE 'awe enabled', 1 RECONFIGURE GO AWE memory cannot be managed dynamically like normal memory in SQL Server. SQL Server takes all the RAM it can automatically when it starts (except for approximately 128 MB, which is reserved for the operating system), but it will not release any of this RAM until SQL Server is restarted. This situation might be fine for a dedicated SQL Server, but if you are running other applications or services on the same server or are running multiple instances of SQL Server on the same server, then you must specify the maximum amount of RAM that SQL Server can take at startup. You can specify this value using the max server memory configuration option. If you change this setting, you need to restart the SQL Server service for the new setting to take affect. To set the maximum amount of memory that AWE memory can access, use the commands that are shown in Example 26-5. Example 26-5 Command sequence to set the maximum memory available to AWE<br />
<br />
SP_CONFIGURE 'max server memory', 4096 RECONFIGURE GO In Example 26-5, we specify that SQL Server should only use 4 GB of RAM, leaving any other RAM available in the server free for other applications. Multiple instances of SQL Server can be used with AWE memory; however, this can present an operational overhead and might defeat the purpose of more server RAM. The goal of using AWE memory should be to support a single, large instance of SQL Server, as opposed to many smaller instances of SQL Server functioning on a single server.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
907<br />
<br />
26.10 SQL Server indexes SQL Server data and index pages are both 8 KB in size. SQL Server data pages include all of the data associated with rows of a table, except text and image data. In the case of text and image data, the SQL Server data page that includes the row associated with the text/image column will include a pointer to a B-tree structure of one or more 8 KB pages that include the text or image data. SQL Server index pages include only the data from columns that comprise a particular index. Thus, index pages effectively compress information that is associated with many more rows into an 8 KB page than an 8 KB data page does. An important I/O performance concept to visualize is that the I/O performance benefit of indexes comes from this information compression. This concept is true if the columns picked to be part of an index form a relatively low percentage of the row size of the table. When an SQL query asks for a set of rows from a table in which columns in the query match certain values in the rows, SQL Server can save I/O operations and time by reading the index pages to look for the values and then access only the rows in the table required to satisfy the query. This method is more efficient than having to perform I/O operations to scan all rows in the table to locate the required rows. This concept is true if the indexes defined are selected well. There are two types of SQL Server indexes, non-clustered and clustered, and both are built upon B-tree structures formed out of 8 KB index pages. The difference is at the bottom of the B-tree structures, which are referred to as the leaf level in SQL Server documentation. The upper parts of index B-tree structures are referred to as nonleaf levels of the index. A B-tree structure built for every single index is defined on a SQL Server table.<br />
<br />
26.10.1 Non-clustered indexes In non-clustered indexes, the leaf level nodes include only the data that participates in the index, along with pointers to locate quickly the remaining row data on the associated data page. In the worst-case scenario, each row access from the non-clustered index requires an additional random disk I/O to retrieve the row data. In a best-case scenario, many of the required rows are on the same data page and thus allow retrieval of several required rows with each data page fetched. Non-clustered indexes are most useful for fetching few rows with good selectivity from large SQL Server tables based on a key value. Non-clustered indexes are B-trees formed out of 8 KB index pages, and the bottom, or leaf level, of the B-tree of index pages includes all the data from the columns that comprised that index.<br />
<br />
908<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
When a non-clustered index is used to retrieve information from a table based on a match with the key value, the index B-tree is traversed until a key match is found at the leaf level of the index. A pointer jump is made if columns from the table are needed that did not form part of the index. This pointer jump likely requires a random I/O operation on the disk. It might even require the data to be read from another disk, if the table and its accompanying index B-trees are large in size. If multiple pointers lead to the same 8 KB data page, less of an I/O performance penalty will be paid because it is only necessary to read the page into data cache once. For each row returned for a SQL query that involves searching with a non-clustered index, one pointer jump is required. These pointer jumps are the reason that non-clustered indexes are better suited for SQL queries that return only one or a few rows from the table. Queries that require a lot of rows to be returned are better served with a clustered index.<br />
<br />
26.10.2 Clustered indexes In clustered indexes, the leaf level nodes of the index are the actual data rows for the table. Therefore, no pointer jumps are required for retrieval of table data. Range scans based on clustered indexes perform well because the leaf level of the clustered index (thus, all rows of that table) is ordered physically on disk by the columns that comprise the clustered index and, due to this fact, perform I/O in 64 KB extents. If there is not a lot of page splitting on the clustered index B-tree (nonleaf and leaf levels), these 64 KB I/Os should be physically sequential. There can only be one clustered index per table. There is a simple physical reason for this. While the upper parts (commonly referred to in SQL Server documentation as nonleaf levels) of the clustered index B-tree structure are organized just like the non-clustered index B-tree structures, the bottom level of the clustered index B-tree are the actual 8 KB data pages associated with the table. There are two performance implications here: Retrieval of SQL data based on key search with a clustered index requires no pointer jump (with a likely nonsequential change of location on the hard disk) to get to the associated data page because the leaf level of the clustered index is already the associated data page. The leaf level of the clustered index is sorted by the columns that comprise the clustered index. Because the leaf level of the clustered index includes the actual 8 KB data pages of the table, this means the row data of the entire table is physically arranged on the disk drive in the order determined by the clustered index.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
909<br />
<br />
This provides a potential I/O performance advantage when fetching a significant number of rows from this table (at least greater than 64 KB) based on the value of the clustered index, because sequential disk I/O is being used (unless page splitting is occurring on this table, which will be discussed later in 26.10.7, “Importance of FILLFACTOR and PAD_INDEX” on page 914). That is why it is important to pick the clustered index on a table based on a column that will be used to perform range scans to retrieve a large number of rows.<br />
<br />
26.10.3 Covering indexes A special situation with non-clustered indexes is the covering index. A covering index is a non-clustered index that is built upon all of the columns required to satisfy an SQL query, both in the selection criteria and the WHERE predicate. Covering indexes can save a huge amount of I/O, and hence improve the performance of a query. However, it is necessary to balance the costs of creating a new index (with its associated B-tree index structure maintenance) against of the I/O performance gain the covering index will bring. If a covering index benefits a query or set of queries that are run very often on SQL Server, creating that covering index might be worth it.<br />
<br />
26.10.4 Automatic covering indexes or covered queries SQL Server Index intersection feature allows the query processor to consider multiple indexes from a given table, build a hash table based on those multiple indexes, and utilize the hash table to reduce I/O for a given query. The hash table that resulted from the index intersection has become, in essence, a covering index and provides the same I/O performance benefits that covering indexes do. Index intersection provides greater flexibility for database user environments in which it is difficult to predetermine all of the queries that will be run against the database. A good strategy to follow in this case would be to define single-column, non-clustered indexes on all columns that are frequently queried and let index intersection handle situations where a covered index is needed.<br />
<br />
910<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
26.10.5 Index selection How indexes are chosen significantly affects the amount of disk I/O generated and, subsequently, performance. The previous sections described why non-clustered indexes are good for retrieval of a small number of rows and clustered indexes are good for range-scans. Here is some additional information: Try to keep indexes as compact (fewest number of columns and bytes) as possible. This is especially true for clustered indexes because non-clustered indexes will use the clustered index as their method for locating row data. In the case of non-clustered indexes, selectivity is important, because if a non-clustered index is created on a large table with only a few unique values, usage of that non-clustered index will not save I/O during data retrieval. In fact, using the index would likely cause much more I/O than a sequential table scan of the table. Some examples of good candidates for a non-clustered index are invoice numbers, unique customer numbers, social security numbers, and telephone numbers. Clustered indexes are much better than non-clustered indexes for queries that match columns or search for ranges of columns that do not have a lot of unique values because the clustered index physically orders the table data, allowing for sequential 64 KB I/O on the key values. Some examples of possible candidates for a clustered index include states, company branches, date of sale, zip codes, and customer district. It would tend to be a waste to define a clustered index on the columns that just have unique values unless typical queries on the system fetch large sequential ranges of the unique values. The key question to ask when trying to pick on which column on each table to create the clustered index is “Will there be a lot of queries that need to fetch a large number of rows based on the order of this column?” The answer is very specific to each user environment. One company might do a lot of queries based on ranges of dates, while another company might do a lot of queries based on ranges of bank branches.<br />
<br />
26.10.6 Clustered index selection Clustered index selection really involves two major decisions: Determine which column of the table will benefit most from the clustered index in terms of providing sequential I/O for range scans. Determine whether to use the clustered index to affect the physical placement of table data while avoiding hot spots. A hot spot occurs when data is placed on hard drives such that many queries are trying to read or write data in the same area of the disks at the same time. A hot<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
911<br />
<br />
spot creates a disk I/O bottleneck, because more concurrent disk I/O requests are being received by that hard disk than it can handle. Solutions to a hot spot are to either stop fetching as much data from this disk or to spread the data across multiple disks to support the I/O demand. This type of consideration for the physical placement of data can be critical for good concurrent access to data among hundreds or thousands of SQL Server users. These two decisions often conflict with each other and the best overall decision will have to balance the two. In high user load environments, improved concurrency (by avoiding hot spots) can often be more valuable than the performance benefit of placing the clustered index on that column. In previous versions of SQL Server, for tables without a clustered index (tables like this are referred to as heaps) inserted rows would always be placed at the physical end of the table on disk. This created the possibility of a hot spot at the end of a very busy table. SQL Server's storage management algorithms provide free space management, which prevents this behavior. Now when rows are inserted in heaps, SQL Server makes use of the PFS (Page Free Space) pages to quickly locate available free space somewhere in the table in which the row can be inserted. PFS pages indicate free space through the table. This recovers deleted space and avoids insertion hot spots because inserts will be spread through the physical disk space throughout the table. Free space management affects clustered index selection. Because clustered indexes affect physical data placement, hot spots can occur when a clustered index physically sequences based on a column where many concurrent inserts occur at the highest column value, which will be located on the same physical disk location. For columns with constantly increasing values, be aware of how a clustered index on that column sequentially orders data rows on disk by that column and remember that by placing the clustered index on another column or by not including a clustered index on the table, this sequential data placement changes to another column or does not take place at all. Here is a common scenario to help illustrate clustered index selection. Suppose a table includes an invoice date column, a unique invoice number column, and other data. Suppose that about 10 000 new records are inserted into this table every day and that SQL queries often need to search this table for all records for one week's worth of data and many users need concurrent access to this table. The invoice number would not be a good candidate for the clustered index for two reasons: Invoice number is unique and users do not tend to search on ranges of invoice numbers, so placing invoice numbers physically in sequential order on<br />
<br />
912<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
disk is not likely to be helpful because range scans on invoice numbers will likely not happen. The values for invoice number likely increase monotonically (1001, 1002, 1003, and so on). If the clustered index is placed on an invoice number, inserts of new rows into this table all happen at the end of the table (beside the highest invoice number) and, therefore, on the same physical disk location, creating a hot spot. Next, consider the invoice date column. To maximize sequential I/O, the invoice date would be a good candidate for a clustered index because users often are searching for one week's worth of data (about 70 000 rows). However, from the concurrency perspective, the invoice date might not be a good candidate for the clustered index. If the clustered index is placed on an invoice date, all data will tend to be inserted at the end of the table, given the nature of dates, and a hot spot can occur on the hard disk that holds the end of the table. Note that the fact that the insertions happened at the end of the table is somewhat offset by the fact that 10 000 rows are inserted for the same date, so the invoice date would be much less likely to create a hot spot than the invoice number. Also, a hardware RAID controller would help spread out the 10 000 rows across multiple disks, which would also help minimize the possibility of an insertion hot spot. There is no perfect answer to this scenario. It might be necessary to decide that it is worth the risk of hot spotting and choose to place the clustered index on the invoice date in order to speed up queries involving invoice date ranges. If this is the case, monitor disk queuing on the disks associated with this table carefully and keep in mind that the inserts can queue up behind each other trying to get to the end of the table. A solution to this scenario would be to define the clustered index on the invoice date because of the benefit to range scans based on the invoice date and so that invoice numbers are not physically sequential on disk. Another way to think about hot spots is within the context of selects. If many users are selecting data with key values that are very close to but not in the same actual row as each other, a majority of disk I/O activity will tend to occur within the same physical region of the disk I/O subsystem. This disk I/O activity can be spread out more evenly by defining the clustered index for this table on a column that will spread these key values evenly across the disk. If all selects are using the same unique key value, using a clustered index will not help balance the disk I/O activity of this table. Use of RAID (either hardware or software) would help alleviate this problem as well by spreading the I/O across many disk drives. The type of behavior described here can be viewed as disk access contention. It is not locking contention.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
913<br />
<br />
26.10.7 Importance of FILLFACTOR and PAD_INDEX If a SQL Server database experiences a large amount of insert activity, it is important to provide and to maintain some open space on index and data pages to prevent page splitting. Page splitting occurs when an index page or data page can no longer hold any new rows and a row needs to be inserted into the page because of the logical ordering of data defined in that page. When this occurs, SQL Server needs to divide up the data on the full page and move about half of the data to a new page so that both pages now have some open space. This consumes some system resources and time. When indexes are initially built, SQL Server places the index B-tree structures on contiguous physical pages, which allows for optimal I/O performance scanning the index pages with sequential I/O. When page splitting occurs and new pages need to be inserted into the logical B-tree structure of the index, SQL Server must allocate new 8 KB index pages. This occurs somewhere else on the hard drive and will break up the physically sequential nature of the index pages. This switches I/O operations from sequential to nonsequential and cuts performance in half. Excessive amounts of page splitting should be resolved by rebuilding the index to restore the physically sequential order of the index pages. This same behavior can be encountered on the leaf level of the clustered index, thereby affecting the data pages of the table. In Performance console, check the counter SQL Server: Access Methods - Page Splits. Non-zero values for this counter indicates that page splitting is occurring and that you should perform further analysis with the DBCC SHOWCONTIG command. The DBCC SHOWCONTIG command is very helpful, and you can use it to reveal whether excessive page splitting has occurred on a table. Scan Density is the key indicator that DBCC SHOWCONTIG provides. It is good for this value to be as close to 100% as possible. If this value is well below 100%, rebuild the clustered index on that table using the DROP_EXISTING option to defragment the table. The DROP_EXISTING option of the CREATE INDEX statement permits re-creation of existing indexes and provides better index rebuild performance than dropping and re-creating the index. The FILLFACTOR option on the CREATE INDEX and DBCC REINDEX commands provides a way to specify the percentage of open space to leave on index and data pages. The PAD_INDEX option for CREATE INDEX applies what has been specified for FILLFACTOR on the nonleaf level index pages. Without the PAD_INDEX option, FILLFACTOR mainly affects the leaf level index pages of the clustered index. It is a good idea to use the PAD_INDEX option with FILLFACTOR. The optimal value to specify for FILLFACTOR depends upon how much new data will be inserted within a given time frame into an 8 KB index and data page. It is<br />
<br />
914<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
important to keep in mind that SQL Server index pages typically include many more rows than data pages because index pages only include the data for columns associated with that index, while data pages hold the data for the entire row. Also bear in mind how often there will be a maintenance window that will permit the rebuilding of indexes to avoid page splitting. Strive toward rebuilding the indexes only as the majority of the index and data pages have become filled with data. Part of what allows this to happen is the proper selection of a clustered index for a given table. If the clustered index distributes data evenly so that new row inserts into the table happen across all of the data pages associated with the table, the data pages will fill evenly. Overall, this will provide more time before page splitting starts to occur and it is necessary to rebuild the clustered index. The other part of the decision is the FILLFACTOR, which should be selected partly on the estimated number of rows that will be inserted within the key range of an 8 KB page for a given time frame and how often scheduled index rebuilds can occur on the system. This is another situation in which a judgment call must be made, based on the performance trade-offs between leaving a lot of open space on pages versus page splitting. If a small percentage for FILLFACTOR is specified, it will leave large open spaces on the index and data pages. This helps avoid page splitting but will also negate some of the performance effect of compressing data onto a page. SQL Server performs faster if more data is compressed on index and data pages because it can generally fetch more data with fewer pages and I/Os if the data is more compressed on the pages. Specifying too high a FILLFACTOR can leave too little open space on pages and allows pages to overflow too quickly, causing page splitting. Before using FILLFACTOR and PAD_INDEX, remember that reads tend to far outnumber writes, even in an online transaction processing system. Using FILLFACTOR will slow down all reads, because it spreads tables over a wider area (reduction of data compression). Before using FILLFACTOR and PAD_INDEX, it is a good idea to use Performance console to compare SQL Server reads to SQL Server writes and to only use these options if writes are a substantial fraction of reads (say, more than 30%). If writes are a substantial fraction of reads, the best approach in a very busy OLTP system is to specify as high a FILLFACTOR as feasible that will leave a minimal amount of free space per 8 KB page but still prevent page splitting and allow the SQL Server to reach the next available time window for rebuilding the index. This methodology balances I/O performance (keeping the pages as full as possible) and page splitting avoidance (not letting pages overflow). This might take some experimentation with rebuilding the index with varying FILLFACTOR values and then simulating load activity on the table to validate an optimal value<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
915<br />
<br />
for FILLFACTOR. When the optimal FILLFACTOR value has been determined, automate the scheduled rebuilding of the index as an SQL Server Task. In the situation where there will be no write activity into the SQL Server database, the FILLFACTOR value should be set at 100% so that all index and data pages are filled completely for maximum I/O performance.<br />
<br />
26.11 SQL Server performance objects After you install SQL Server, the following objects are available in Performance console that are available from a Windows 2003 operating system by running perfmon in a command prompt: <br />
<br />
916<br />
<br />
SQLServer Access methods SQLServer Backup device SQLServer Buffer manager SQLServer Cache manager SQLServer Databases SQLServer General statistics SQLServer Latches SQLServer Locks SQLServer Memory manager SQLServer Replication agents SQLServer Replication dist SQLServer Replication logreader SQLServer Replication merge SQLServer Replication snapshot SQLServer SQL statistics SQLServer User settable<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
SQL Server provides a performance chart (SQLCTRS.PMC) to monitor SQL Server-specific activity. SQLCTRS.PMC include the counters that are listed in Table 26-4. Table 26-4 SQL Server SQLCTRS.PMC counters<br />
<br />
SQL Server Object<br />
<br />
Counter<br />
<br />
Description<br />
<br />
SQL Server Buffer Manager<br />
<br />
Buffer Cache Hit Ratio<br />
<br />
Percentage of data pages that were retrieved from the buffer cache<br />
<br />
SQL Server General Statistics<br />
<br />
User Connections<br />
<br />
The number of SQL Server users<br />
<br />
SQL Server Memory Manager<br />
<br />
Total Server Memory (KB)<br />
<br />
The total amount of memory that SQL Server is using<br />
<br />
SQL Server SQL Statistics<br />
<br />
SQL Compilations/sec<br />
<br />
The number of compilations processed by SQL Server<br />
<br />
SQL Server Buffer Manager<br />
<br />
Page Reads/sec<br />
<br />
Number of physical database page reads<br />
<br />
SQL Server Buffer Manager<br />
<br />
Page Writes/sec<br />
<br />
Number of physical database page writes<br />
<br />
See Chapter 19, “Analyzing bottlenecks for servers running Windows” on page 655 for Performance console counters to watch when detecting performance bottlenecks. Table 26-5 lists specific counters to watch. Table 26-5 SQL Server Performance Monitor counters<br />
<br />
Counter<br />
<br />
Description<br />
<br />
PhysicalDisk: Avg. Disk Queue Length<br />
<br />
Hard drives that have I/O requests queued to them waiting to be processed will degrade response times to users. If this value is consistently over two times the number of disks in the array (for example, 8 for a 4-disk array), you should increase the number of disks or upgrade the disk subsystem. On hard drives that include the log files, disk queuing is not a useful measure because SQL Server log manager does not queue more than a single I/O request to the log files. Note: You will need to run DISKPERF -Y and restart Windows NT before monitoring these counters. By default, this DISKPERF command is not required for Windows 2000.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
917<br />
<br />
Counter<br />
<br />
Description<br />
<br />
System: Processor Queue Length<br />
<br />
If this value is non-zero, the processors are receiving more work requests than they can handle as a group. In a normal server, this would actually be reasonable — the CPUs will probably always outpace the other subsystems. If the CPU utilization is low, this could indicate a bottleneck in another subsystem. A good processor queue length for SQL Server is 2x (number of processors). Anything greater than this needs investigation and could be indicative of a need for a CPU subsystem upgrade. Several different activities could be contributing to processor queuing. Eliminating hard and soft paging helps save CPU resources. Other methods that help reduce processor queuing include SQL query tuning, picking better SQL indexes to reduce disk I/O (and, thus, CPU), or adding more CPUs (processors) to the system<br />
<br />
Memory: Pages/sec<br />
<br />
If this value is non-zero, hard page faults are occurring as Windows is paging memory to disk. Either reduce the memory given to SQL Server or add more RAM to the server.<br />
<br />
Memory: Page Reads/sec<br />
<br />
If this value is greater than 5, add RAM to the server or remove other applications.<br />
<br />
Memory: Page Faults/sec<br />
<br />
If this value is non-zero, soft page faults are occurring. This means there are applications on the server requesting memory pages still in RAM but out of the Windows Working Set. On a dedicated SQL Server machine, SQL Server is likely to be the cause, and can be verified by examining the counter Process: Pages Faults/sec.<br />
<br />
Process: Page Faults/sec<br />
<br />
Examine this for the SQLSERVR process. If it is non-zero, then SQL Server is causing soft page faults. To prevent these from happening, set SQL Server’s working set to be exactly the same as SQL Server memory allocation. Execute the following command in ISQL/W: sp_configure ‘set working set size’, 1 Note: Ignore all soft faults that occur as SQL Server starts up and the data cache is first being exercised, as this is normal.<br />
<br />
LogicalDisk: %Disk Time; Processor: %Processor Time; System: %Total Processor Time<br />
<br />
If the Processor: %Processor Time counter chart is consistently high (greater than 70%) and the LogicalDisk: % Disk Time counter is consistently low, this indicates a CPU-bound state. If the Processor: %Processor Time chart is consistently low (less than 50%), and the LogicalDisk: % Disk Time counter is consistently high, this indicates an I/O bound state.<br />
<br />
SQLServer: Cache Hit Ratio<br />
<br />
The percentage of time that a request was found less than 90%, adding more memory can improve the hit ratio and thereby performance because additional memory can mainly be used for additional SQL Server data cache.<br />
<br />
SQLServer: I/O-Transactions/ sec<br />
<br />
The number of Transact-SQL command batches executed per second. High Transactions/sec means good throughput.<br />
<br />
918<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Counter<br />
<br />
Description<br />
<br />
SQLServer: I/O-Pages Reads/sec<br />
<br />
The number of physical page reads per second.<br />
<br />
SQLServer: I/O Single Page Writes/sec<br />
<br />
The number of single page writes performed per second by logging and cache flushes. Reducing single-page write activity is important for optimal tuning. One way is to do this is to ensure that you do not run out of free buffers in the free buffer pool. If you do, single page writes will occur when waiting for an unused cache buffer to flush.<br />
<br />
SQLServer: User Connections<br />
<br />
The number of user connections. As a general rule, set the number of user connections to the expected number of concurrent users + 10.<br />
<br />
Note: The SQLServer statistics only display when the SQL Server is running. Stopping and restarting the SQL Server interrupts and resumes the display of SQL Server statistics automatically.<br />
<br />
26.11.1 Other diagnostic and performance tools This section discusses the tools that are useful for troubleshooting and tuning performance.<br />
<br />
SQL Trace and SQL Profiler SQL Trace and SQL Profiler are related tools. SQL Trace is the name for the tracing facility that is configured by using a set of system stored procedures. This facility does not drop trace events when the server is under stress (unlike SQL Profiler). SQL Profiler is a GUI application front end for SQL Trace. SQL Trace and SQL Profiler is a good tool to use to find poorly performing queries, to determine the cause of deadlocks, or to collect a sample workload to replay for stress testing. A trace template can be used to collect workloads in the format needed by the Database Engine Tuning Advisor.<br />
<br />
Database Engine Tuning Advisor Database Engine Tuning Advisor (DTA) replaces the Index Tuning Wizard from SQL Server 2000. DTA recommends changes in indexes and partitioning for an existing database based upon a typical workload (a SQL trace) that is collected from the production system. You can use DTA to identify indexes that are not being used by your application. These indexes can be dropped because their maintenance creates unnecessary overhead. You can use DTA to identify new indexes that currently do not exist. These indexes can be added, potentially reaping large performance benefits.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
919<br />
<br />
Dynamic management views One of the design goals of SQL Server 2005 is for users to be able to effectively troubleshoot and tune their databases. Dynamic management views provide this visibility to the internal workings of SQL Server 2005. This visibility is not provided with SQL 2000 or earlier versions of SQL Server. The server-wide dynamic management views are: sys.dm_db_ For databases that provide space and usage information by file, index, partition, task, and session. Note that queries that return index fragmentation statistics can cause intensive I/O on that index. sys.dm_exec_ For execution related information, including query plans, cursors, and sessions. sys.dm_io_ Provides I/O information for data and log files, pending I/O information, and tape status. sys.dm_os_ For SQLOS related information, including memory, scheduling, tasks and wait statistics. sys.dm_tran_ For transaction-related information including active transactions, locking, snapshots, and the version store. Note that selecting row version information could return many rows and be resource intensive. The following are the component specific dynamic management views: <br />
<br />
sys.dm_clr_ for the Common Language Runtime feature sys.dm_db_mirroring_ for the Database Mirroring feature sys.dm_fts_ for the Full Text Search feature sys.dm_qn_ for the Query Notifications feature sys.dm_repl_ for the Replication feature sys.dm_broker_ for the Service Broker feature<br />
<br />
Catalog views expose static metadata that describes all the user viewable objects in a SQL Server instance. For example, sys.master_files gives all the database file names that are known to the SQL Server instance. By combining catalog views with dynamic management views, you can create queries that interpret the internal data for troubleshooting and performance analysis, as shown in Example 26-1 on page 884.<br />
<br />
920<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
SQLdiag The SQLdiag utility is a general purpose diagnostic tool. It collects performance logs, event logs, SQL traces, SQL blocking data, and SQL configuration data. An especially useful mode for using this tool is to start it, reproduce an issue, and shut it down. The captured information that results from this mode can be analyzed for troubleshooting. You can extensively customize the data SQLdiag collects.<br />
<br />
Query hints and plan guides Query hints are provided with SQL 2000 and SQL 2005. They can be used to force the query optimizer to choose a specific query plan. They are useful when the optimizer occasionally does not choose the most efficient plan. Plan guides are an extension of this. Using a plan guide, it is possible to modify a query, providing query hints, without changing the text of the query, which is useful when you have a third-party application and do not want to or are unable to modify the code.<br />
<br />
Chapter 26. Microsoft SQL Server<br />
<br />
921<br />
<br />
922<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
27<br />
<br />
Chapter 27.<br />
<br />
Oracle Oracle tuning is a very complex topic. According to J. S. Couchman, author of Oracle 8 Certified Professional: DBA Certification Exam Guide (ISBN 0072130601), Oracle tuning is a five-step process: 1. Application tuning Quite often, the most important step is the first one. Poorly written SQL queries and applications can dramatically reduce the performance of the system. This topic is very complex and requires a good understanding of the SQL language and RDBMS mechanism. See Chapter 17 of Oracle 8 Certified Professional: DBA Certification Exam Guide, and Oracle 9i. Database Performance Guide, Part 1 (Oracle Corporation), for information about how to detect poorly performing SQL instructions. 2. Operating system tuning Having optimized the SQL code, the next step is operating system configuration and tuning. Windows 2000 and Windows Server 2003 are largely self-tuning operating systems but a few operations should be performed. These operations are described in 27.4, “Operating system optimization” on page 931. 3. Oracle memory tuning Much can be done to improve Oracle instance memory internal organization. Improving the distribution of memory resources among the dictionary cache,<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
923<br />
<br />
the database buffer cache and the redo log buffer cache can have spectacular results on the DBMS performance. 4. Oracle I/O tuning As with any other software application, the I/O is a very frequent bottleneck. A better choice of file distribution, as well as a more clever choice of RAID level, can make the difference between a DBMS that works well and one that works poorly. 27.6, “Oracle disk I/O optimization” on page 937 provides many basic recommendations on how to improve I/O performance. 5. Resource contention tuning Lastly, resource contention can be the cause of poor performance. Deadlocks, that is, mutual locks between processes that stop both of them, are an example of a situation in which resource contention can dramatically reduce performance. You can find a detailed discussion about resource contention tuning in Chapter 20 of Oracle 8 Certified Professional: DBA Certification Exam Guide. You can find a list of publications that discuss Oracle tuning in “Related publications” on page 1021. This chapter provides introductory information about Oracle tuning and includes the following topics: <br />
<br />
27.1, “Oracle architecture” on page 924 27.2, “OLTP versus OLAP” on page 929 27.3, “Important subsystems” on page 931 27.4, “Operating system optimization” on page 931 27.5, “Oracle memory optimization” on page 932 27.6, “Oracle disk I/O optimization” on page 937 27.7, “Monitoring DBMS performance” on page 946 27.8, “Oracle Performance Monitor for Windows 2000” on page 947 27.9, “Summary” on page 949<br />
<br />
27.1 Oracle architecture There are countless books that can provide an in-depth description of the Oracle architecture. In this section, we describe a basic overview of the Oracle architecture. There are three basic components to Oracle (Figure 27-1): The memory space The Oracle processes The DBMS files<br />
<br />
924<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Clients DBMS instance Disp<br />
<br />
Foreground processes<br />
<br />
Server<br />
<br />
User proc System global area Shared pool<br />
<br />
Library cache<br />
<br />
Process monitor<br />
<br />
Redo log buffer cache<br />
<br />
Dictionary cache<br />
<br />
System monitor<br />
<br />
Log writer<br />
<br />
Program global area<br />
<br />
Database buffer cache<br />
<br />
Archiver<br />
<br />
Check point<br />
<br />
DB writer<br />
<br />
Memory<br />
<br />
Background processes<br />
<br />
DBMS files Initialization files<br />
<br />
Control files<br />
<br />
Redo log files<br />
<br />
Archived redo log files<br />
<br />
Data files<br />
<br />
Figure 27-1 Oracle database structure<br />
<br />
27.1.1 Memory The memory addressed by an Oracle instance is composed of two main areas: the system global area (SGA) and the program global area (PGA). The SGA is a shared memory that is accessed by all background and foreground Oracle processes. The PGA is the area where server processes work on behalf of users to record private (non-shared) information. The session information such as the path of a query is saved in this area. The SGA consists of three areas: 1. Redo log buffer cache Each server process writes a log of data changes. In order to improve I/O performance, the server processes only write to the redo log buffer cache.<br />
<br />
Chapter 27. Oracle<br />
<br />
925<br />
<br />
When the SQL Commit statement is used, the log writer process then moves the content of the cache to the redo log files. 2. Database buffer cache If the data a server process requires is in the database buffer cache, it is read from here. If not, it is read into the buffer cache by the server process. The server process only writes to the database buffer cache. An asynchronous background process (the database writer) asynchronously updates the data files. 3. Shared pool The shared pool memory area is composed of two distinct caches: the dictionary cache and the library cache. – Dictionary cache While conventional file processing environments consist of non-self-describing data, a database is a set of self-describing data. Data describing other data is known as metadata and is included in the so-called system catalog, also known as the data dictionary. The basic tables included in the data dictionary are: DBA_TABLES DBA_TAB_COLUMNS DBA_USERS DBA_VIEWS<br />
<br />
The table of all the tables The table of all the columns The table of all the users The table of all the views<br />
<br />
The data dictionary is included in the SYSTEM tablespace. Because each select statement as well as each DML (Data Manipulation Language) instruction needs to check the data dictionary, a lot of I/O is to be expected on the disks that are included in the data dictionary. The dictionary cache avoids many of the I/O operations by caching the content of the data dictionary. Tip: In DBMS theory, the term DML is used to label all the SQL operations that can alter data (for example, INSERT, DELETE, and UPDATE). – Library cache The execution of an SQL statement consist of three main steps: i. Parse statement ii. Execute statement iii. Fetch values During the first phase the SQL statement is parsed, which means that an execution plan is created. At the end of the parsing phase the parse tree is<br />
<br />
926<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
saved in the library cache. Subsequent statements having the same SQL structure will omit this phase by loading the parse tree directly from the library cache. During the second phase the execution plan is executed and then during the third phase returned data is stored in a cursor. A similar but slightly more complex process takes place when DML instructions are processed. Starting with release 9i, Oracle can re-size the SGA without shutting down the instance. The same holds for the size of the buffer cache and the shared pool. Another important feature introduced with release 9i is the PGA automatic memory management for instances working in dedicated mode. You can find in-depth descriptions of both these new features in Oracle 9i. Database Concepts. Release 1 (9.0.1).<br />
<br />
27.1.2 Processes Oracle processes are either foreground and background processes. Server processes are used to act on behalf of clients because access to the SGA is allowed to server processes only and not to user processes. Two architectures are possible: Dedicated server processes: for each user process, a dedicated server process is created. Multi-threaded server: a pool of server processes is created. User processes request a connection to one dispatcher process. The dispatcher process puts the request in the server processes queue. When a server process becomes free, the request is taken from the queue and executed. The number of server processes can be automatically altered by the Oracle RDBMS. Server processes, also known as foreground processes, read data from the data files on disk into the database buffer cache in memory. Any further requests for the same data are resolved from the cache, reducing the need for further I/O. The Oracle background processes include (see Oracle 9i. Database Concepts for further information): Log writer (LGWR). The log writer writes sequentially batches of modified entries in the SGA’s redo buffer for one or more transactions to the online redo logs files. Archiver (ARCn) - optional. The archiver processes copy the online redo log files to the archive log files. While a single archiver process (ARC0) is usually adequate, it is possible to activate up to ten archiver processes. The ARCn process is active only if the database works in ARCHIVELOG mode. Database writers (DBWn). These processes write modified database blocks from the SGA database buffer cache to the database files. Although one<br />
<br />
Chapter 27. Oracle<br />
<br />
927<br />
<br />
database writer is usually enough (DBW0), up to 10 database writers can be activated. System monitor (SMON). The SMON process takes care of instance recovery in case of system crash. Process monitor (PMON). The process monitor is responsible for process recovery in case of process crash. Checkpoint (CKPT) - optional. The checkpoint process notifies the DBWn process(es) that updates to the data and control files must be completed to the physical files. It also takes care to update data files and control files so that they include information about the most recent checkpoint. Recovery (RECO) - optional. The RECO process recovers transactions from a distributed database configuration. Lock Management Server (LMS). The LMS process is used for instance recovery in a Oracle 9i Real Application Cluster. Queue Monitor (QMNn) - optional. The QMNn processes are optional processes monitoring message queues for Oracle Advanced Queueing.<br />
<br />
27.1.3 DBMS files The third component of the Oracle architecture includes the actual physical files associated with a database instance. There are the database data files, the online redo log files and the control file. Data files Database data files include tables and indexes. Redo logs The redo log files are the journal of the Oracle database. All changes made to the user or system objects are recorded through the redo logs. The redo logs are also used for error recovery in case of media failure or system crash. Each Oracle instance requires at least two redo log files and for data security, they should be located on a fault-tolerant RAID device. Control files The control file includes information pertinent to the database itself. It includes information such as the name of the database, date and time stamp of creation, location of database files and redo logs, and synchronization information. Without this file, the database will not start properly. Initialization files There is one additional file that is important within the Oracle instance. It is the init<sid>.ora file. The init<sid>.ora file is read at database startup. It<br />
<br />
928<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
controls the execution environment for the database being started. It includes information such as the location of the control file, number of rollback segments, and the initial values for Oracle’s tunable parameters. There are over 200 parameters in the init<sid>.ora that affect the performance and function of the Oracle database. The init<sid>.ora file is located into the ORACLE_BASE\admin\<sid>\pfile directory, where ORACLE_BASE is the root of the Oracle directory tree. The name of the initialization file init<sid>.ora is set into the parameter value ORA_SID_PFILE inside the HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\HOMEID registry key.<br />
<br />
27.2 OLTP versus OLAP A database is a self-describing collection of data managed by means of a software product (database management system, or DBMS) for a specific purpose. Among the different possible uses of a database, two are particularly relevant from both a commercial and architectural point of view: Transactions recording, known as online transaction processing or OLTP Decision support, known as online analytical processing or OLAP While the data structures are quite often the same (that is, they are both relational databases), the technical structure of an OLTP system is very different from that of an OLAP one. This section introduces these differences to help the reader better understand the Oracle tuning topics discussed in the next sections.<br />
<br />
27.2.1 Online transaction processing OLTP systems are transaction recording systems. A transaction is an indivisible unit of work, which means that all the steps of the transaction must be completed. If, for some reason, some part of the transaction cannot be executed, all the previously executed steps must be rolled back. To allow Oracle to roll back the transaction, all the data changes must be logged. Typical operations are updates concerning a small number of small tables. Often, real OLTP systems also have long batch processes reading big amounts of data in order to create reports, but this is not the true purpose of the system. OLTP systems can have in excess of thousands of tables; for example, SAP R/3 can have 10 000 to 20 000 tables. Sometimes, some of these tables can grow and become very large, but this is not by design. The database is normalized so tables are typically small. The main concern is data integrity. A two-phase locking protocol is used in order to serialize transactions, thereby avoiding deadlocks, dirty reads, unrepeatable<br />
<br />
Chapter 27. Oracle<br />
<br />
929<br />
<br />
reads, and phantoms. See Database Models, Languages and Design, by James L. Johnson, Oxford University Press, ISBN 0195107837, for more details.<br />
<br />
27.2.2 Online analytical processing OLAP systems are analytical systems, meaning that the typical user activity is not data entry but data analysis. Hence data integrity is not as important as it is for OLTP systems. What is important is providing the users with an efficient and flexible tool for in-depth data manipulation. The OLAP acronym has been introduced in Providing OLAP to User-Analysts: An IT mandate by E. F. Codd, S. B. Codd and C. T. Salley. According to the authors, an OLAP software should have the following twelve properties: <br />
<br />
Multidimensional conceptual view Transparency, accessibility Consistent reporting performance Client/server architecture Generic dimensionality Dynamic sparse matrix handling Multi-user support Unrestricted cross-dimensional operations Intuitive data manipulation Flexible reporting Unlimited dimensions Aggregation levels<br />
<br />
The intrinsic nature of OLAP operations is multidimensional. Queries such as “I want to know how many System x servers have been sold in North Carolina in April” are three-dimensional queries whose dimensions are products (System x servers), state (North Carolina) and time (April). Relational databases have not been built for multidimensional queries. These queries need many joins; joins of more than five tables typically need too much time. From many points of view, the best DBMS technology for OLAP is that of multidimensional databases (MDDBs) such as Oracle Express. Data is recorded in an operational system (an Oracle relational database OLTP system) and then transferred to the multidimensional database (Oracle Express). Satisfactory performance can be obtained by performing analysis directly on an MDDB, but data is only as recent as the latest data loading from the operational system. A more flexible solution is to move operational system data to a new relational database tuned for OLAP activity, the data warehouse or data mart. Data warehouses typically consist of a very large table (fact table) and some smaller tables (dimension tables). Typical operations are joins between the fact table and the dimension tables.<br />
<br />
930<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
User activity consists of few operations reading large amounts of data. It is a mistake, however, to tune the system only for reading activity. During data loading, data must be written in Oracle data files, indexes must be changed, and sometimes logs must be recorded. If no time constraints exist for loading, it is acceptable to tune the system for pure reading, but if large amounts of data must be loaded often in a short time, it is important to design the system so that I/O is also efficient during writing.<br />
<br />
27.3 Important subsystems The Oracle database server requires a lot of CPU power and large amounts of memory to maximize performance of applications that are computation intensive. An efficient disk subsystem is also very important, especially when handling many random I/O requests. Its primary function is to search and retrieve data from the disk subsystem as requested by client queries. The important subsystems to look at are: Processor Memory Disk subsystem Refer to Chapter 19, “Analyzing bottlenecks for servers running Windows” on page 655 for more information about detecting and removing bottlenecks in these subsystems. As you identify which subsystems are potential sources of bottlenecks, you will have a general idea of what initial actions to take regarding performance optimization.<br />
<br />
27.4 Operating system optimization This section describes how to optimize the Windows 2000 or Windows Server 2003 operating system for the Oracle 9i RDBMS. Further information is provided in Oracle 9i. Database Administrator’s Guide. Release 1 (9.0.1) for Windows by Oracle. The following settings should be made in Windows 2000 and Windows Server 2003 to optimize the performance of Oracle: Reduce foreground applications boost to none as described in 11.2, “Windows Server 2003, 64-bit (x64) Editions” on page 298. Because the DBMS bypasses the file system cache, it does not make sense in a dedicated Oracle server to optimize the system for file system cache<br />
<br />
Chapter 27. Oracle<br />
<br />
931<br />
<br />
usage. Consequently, it is recommended to set the server optimization to maximize data throughput for network applications as described in 11.6, “File system cache” on page 309. Remove any networking protocols not used by Oracle (for example NetBEUI). See 11.8, “Removing unnecessary protocols and services” on page 318 for details on how to perform this operation. Adjust the protocol binding order so that the protocol with the most use is at the top for both server and workstations.See 11.9, “Optimizing the protocol binding and provider order” on page 320 for details on how to perform this operation. If the server is a dedicated database server only, you can stop unnecessary services. At a minimum, you will need the following services: Alerter, Computer Browser, EventLog, Messenger, OracleService<SID>, Oracle<HOME_NAME>TNSListener, OracleStart<SID>, Remote Procedure Call (RPC) Service, Server, Spooler, TCP/IP NetBIOS Helper, Workstation. The following services can be disabled: License Logging Service, Plug and Play, Remote Access Autodial Manager, Remote Access Connection Manager, Remote Access Server and Telephony Service. Do not use any OpenGL screen savers on the server or any other CPU-intensive screen saver, as it consumes otherwise useful CPU power that Oracle can use. Span the Windows Paging File across several disk arrays. Non-recommended or useless tuning operations are: The alteration of Oracle processes priorities is not recommended. The binding of Oracle processes to specific CPUs is not recommended The dynamic change of memory resources is not supported.<br />
<br />
27.5 Oracle memory optimization Oracle instance memory tuning is one of the areas where small parameter changes can produce a big increase, as well as a big decrease, of performance. In this section, we describe shared pool, redo log buffer, and database buffer tuning.<br />
<br />
932<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
27.5.1 Shared pool The shared pool is composed of two areas: the library cache and the dictionary cache.<br />
<br />
Library cache The library cache is composed of three memory areas: Shared SQL area: includes the execution plans of parsed SQL statements PL/SQL programs: includes PL/SQL programs in compiled forms Locks Oracle dynamically tunes the relative size of these areas. The only manually tunable parameter is the shared pool global size variable SHARED_POOL_SIZE. To see if the library cache is properly sized the following simple SQL instruction can be used: select round((sum(pins-reloads)/sum(pins))*100,2) as hit_ratio from v$librarycache; The term pins refers to the number of times a parsed SQL statement was looked for in the library cache. The reloads are the number of times the search was unsuccessful. The library cache hit ratio should be as high as 99%. If the hit ratio is lower, either the instance has been recently started, so the cache is suboptimal, or the shared pool is insufficient and the size should be made larger.<br />
<br />
Dictionary cache The following simple SQL instruction shows if the size of the dictionary cache is optimal select round((sum(gets-getmisses)/sum(gets))*100,2) as hit_ratio from v$rowcache; The term gets refers to the number of times a request was made for informations in the dictionary cache, while the term getmisses refers to the number of unsuccessful requests. The dictionary cache hit ratio should be as high as 99%. If the hit ratio is lower, either the instance has been recently started, so the cache is suboptimal, or the shared pool is insufficient and the size should be made larger.<br />
<br />
27.5.2 Database buffer cache Server foreground processes read from data files into the database buffer cache so the next readings need no I/O operation. Server processes also write modified data into the database buffer cache. Asynchronously, a dedicated background process (DBWn) will move dirty data from the cache to the data files. This way<br />
<br />
Chapter 27. Oracle<br />
<br />
933<br />
<br />
I/O performance is greatly increased. The performance benefits obviously depend on cache hits, that is on how many times server processes looking for data find them in the cache. This section describes the internal structure of the buffer cache and how to tune it.<br />
<br />
Buffer cache architecture The buffer cache is composed of as many buffers as the value of the init<sid>.ora parameter DB_BLOCK_BUFFERS. The size of the buffers are identical and correspond to the init<sid>.ora parameter DB_BLOCK_SIZE. The buffer cache is filled in by foreground processes reading data from data files and flushed out by the DBWn process when one of the following events happen: DBWn time-out (each three seconds) Checkpoint No free buffer Data is removed from the buffer cache according to a least recently used algorithm. Moreover, in order to avoid cache quality worsening due to single full table scan instructions, Table Access Full operations are always put at the end of LRU lists.<br />
<br />
Optimal buffer cache In order to see if the size of the buffer cache is optimal, the following query can be used: select name, value from v$sysstat where name in (‘db block gets’, ‘consistent gets’, ‘physical reads’); Given the output of this select command, the buffer cache hit-ratio can be obtained with the following simple calculation: dbblockgets + consistentgets – physicalreads ------------------------------------------------------------------------------------------------------------------------------------ × 100 ( dbblockgets + consistentgets )<br />
<br />
Enlarging the buffer cache The buffer cache hit-ratio should be 90% or higher. Values between 70% and 90% are acceptable in case it is necessary to resize the buffer cache to improve the library or dictionary hit cache ratios. If the buffer cache hit-ratio is too low, the<br />
<br />
934<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
optimal number of buffers to add to the cache can be obtained with the complex query on the V$DB_CACHE_ADVICE view shown in Example 27-1. Example 27-1 Complex query on the V$DB_CACHE_ADVICE view<br />
<br />
column size_for_estimate format 999,999,999,999 heading ’Cache Size (m)’ column buffers_for_estimate format 999,999,999 heading ’Buffers’ column estd_physical_read_factor format 999.90 heading ’Estd Phys|Read Factor’ column estd_physical_reads format 999,999,999 heading ’Estd Phys| Reads’ SELECT size_for_estimate, buffers_for_estimate, estd_physical_read_factor, estd_physical_reads FROM V$DB_CACHE_ADVICE WHERE name = ’DEFAULT’ AND block_size = (SELECT value FROM V$PARAMETER WHERE name = ’db_block_size’) AND advice_status = ’ON’; Before running the query, you need to activate the V$DB_CACHE_ADVICE view using the following command: alter system set DB_CACHE_ADVICE=ON; Moreover, in order to obtain significant results, you need to be running a representative workload on the system for a reasonable time interval, thereby achieving a stable buffer population. Because the activation of the V$DB_CACHE_ADVICE view has a (minor) impact on CPU load and memory allocation, at the end of the analysis, it is recommended to de-activate the view with the following command: alter system set DB_CACHE_ADVICE=OFF; The output of the query is a set of lines showing the incremental benefits of the various cache sizes. The first column of the query output includes the various cache sizes while the latter column shows the physical reads. Upon increasing the cache size, the physical reads decrease, but the incremental benefit also decreases.<br />
<br />
Chapter 27. Oracle<br />
<br />
935<br />
<br />
Example 27-2 is taken from a small demonstration system. Example 27-2 Example from a demonstration system<br />
<br />
Estd Phys Cache Size (m) Buffers Read Factor Estd PhysReads ---------------- ------------ ----------- -------------3 786 1.59 8 6 1,572 1.00 5 9 2,358 1.00 5 12 3,144 1.00 5 15 3,930 1.00 5 18 4,716 1.00 5 21 5,502 1.00 5 25 6,288 1.00 5 28 7,074 1.00 5 31 7,860 1.00 5<br />
<br />
Multiple buffer pools Starting with Oracle 8, you can create and size multiple buffer pools separately. The database buffer cache is composed of the following three buffer pools: Keep pool Recycle pool Default pool The keep pool stores data that must not be moved out of the buffer cache. The recycle pool is for data that must be moved out quickly of the buffer cache when no longer necessary. Everything else is in the default pool. Unlike the shared pool, whose internal memory areas (library cache, dictionary cache) cannot be separately sized, it is possible to size the keep pool and the recycle pool, so also, as a result, the default pool. Example 27-3 shows how to size a 1000-buffers buffer cache so that 50% is used for the recycle pool, 25% for the keep pool, and 25% for the default pool. Example 27-3 Sizing a 1000-buffers buffer cache<br />
<br />
DB_BLOCK_BUFFERS=1000 DB_BLOCK_LRU_LATCHES=20 BUFFER_POOL_RECYCLE=(buffers:500, lru_latches:10) BUFFER_POOL_KEEP=(buffers:250, lru_latches:5) Latches are memory locks and should be sized according to the following rule: 1 latch for each 50 buffers, as in the previous example.<br />
<br />
936<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
27.5.3 Redo log buffer cache Each server process that updates data first needs to update the redo log files. To improve performance, server processes only write redo log entries into the redo log buffer cache, while the LGWR process is responsible for moving dirty buffers from memory to disks. To avoid buffer data corruption, a locking mechanism (latch) is used, so that only one process at a time can write on the redo log buffer cache. Given the sequential nature of redo log data, only one redo log allocation latch is made available to Oracle server processes. As a result, redo log buffers can be a source of delay, due to high resource contention. In order to see if there is an excessive redo log buffer contention, you can use the following query: select name, value from v$sysstat where name=’redo buffer allocation retries’;. Any value different from 0 shows that processes had to wait for space in the redo log buffer cache. The size of the redo log buffer can be configured by changing the LOG_BUFFER parameter in the init<sid>.ora file. This parameter gives the value in bytes of the cache and must be a multiple of DB_BLOCK_SIZE. Each server process wanting to write to the redo log buffer cache must first get the redo allocation latch. The process will then write as many bytes as allowed by the LOG_SMALL_ENTRY_MAX_SIZE parameter in init<sid>.ora. When this number of bytes has been written, the process must release the latch in order to allow other processes to have a chance of acquiring the lock. To increase the ability of server processes to work concurrently, it is recommended that you size the LOG_SMALL_ENTRY_MAX_SIZE parameter as small as possible.<br />
<br />
27.6 Oracle disk I/O optimization With the introduction of Storage Area Networks, the I/O subsystem tuning has reached a new level of complexity. Indeed, the introduction of Storage Area Networks usually entails: Storage sharing, namely different nodes read and write onto the same storage subsystem with different access patterns. I/O subsystem increased complexity, by virtue of the presence on the I/O communication path of several heterogeneous devices, for example SAN switches and routers, shared among several nodes.<br />
<br />
Chapter 27. Oracle<br />
<br />
937<br />
<br />
Both the aforementioned aspects heavily affect the system performance, but to keep the length of the chapter reasonable, as well as to avoid useless repetitions of information provided elsewhere in this book, we ignore in the following the storage sharing and SAN complexity issues. Before starting the detailed analysis of the I/O subsystem, it is important to emphasize that design is not the search of the best configuration, but the search of the best trade-off. For instance, we can easily improve performance by delaying checkpointing as much as possible, but at the cost of making the Oracle instance recovery very long. Similarly, RAID 0 arrays deliver optimal I/O performance, but with no data protection. The I/O subsystem design is mainly a trade-off between cost, dependability and performance. If we ignore the extra cost of FC SAN devices, the main factors affecting cost are: The base manufacturing cost of the disk. The disk electronic interface (Fibre Channel versus SCSI). The disk size. For instance, one 72 GB disk is a less expensive solution than two 36 GB disks, for the base manufacturing cost does not depend linearly on the disk size. Moreover, the more disks we use, the more enclosures, cabling, racks and so on have to be purchased. The RAID level. Indeed the RAID level choice has a clear impact on the number of disks necessary to host a DB of a given size, thereby impacting the overall cost as explained in the previous point. If we ignore the specific dependability characteristics of Fibre Channel SANs, the main factors affecting dependability are: The disk mean time to failure (MTTF). The number of disks. From a statistical point of view, the higher the number of disks, the higher the risk of a storage subsystem failure. The capacity of each disk. The smaller the capacity, the higher the number of disks. The RAID level. The RAID level has a complex impact on the overall storage subsystem dependability, which we describe in 27.6.2, “The optimal RAID level” on page 940.<br />
<br />
938<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Again, if we ignore the influence of SAN on performance, the main factors affecting performance are: <br />
<br />
The RAID controllers cache size The RAID levels (for example, RAID 10 versus RAID 5) The RAID arrays stripe unit size The Oracle DB block size The tablespaces distribution<br />
<br />
In the following sections, we describe each of these.<br />
<br />
27.6.1 RAID controllers cache size Depending on the storage access patterns, the RAID controller cache can have a major impact on system performance. The cache plays a particularly relevant role for write operations on RAID 5 arrays. Write buffering makes the RAID controller acknowledge the write operation before the write goes to disk. This can positively affect performance in several ways: Updates can overwrite previous updates, thereby reducing the number of disk writes. Grouping several requests, the disk scheduler can achieve optimal performance. By grouping sequential write requests in the controller cache, small writes (operations updating a single stripe unit) can be converted into large writes (full stripe write operations updating all the stripe units of a stripe). A small write request based on read-modify-write algorithm requires two read operations and two write operations just to modify a single stripe unit, while a large write requires a maximum of 2xN operations to update N blocks (N is the number of disks), thereby achieving a much better performance. The read-modify-write algorithm requires two read operations for each stripe unit modification. Even more read operations might be necessary if the RAID controller uses a full XOR algorithm, except in a three disk array. The RAID controller cache can include this data, thereby reducing the number of read operations. However, the relevance of cache size should not be over estimated. Whereas the existence of a small cache can deliver high benefits, often the marginal benefits of doubling the cache size are minimal. Moreover, it is important to remember that the RAID controller cache is only one the many caches affecting I/O subsystem performance. A major role is played by the Oracle DBMS caches. You can find further information about the performance impact of the RAID controller cache size in 9.6.9, “RAID adapter cache size” on page 216 and in the<br />
<br />
Chapter 27. Oracle<br />
<br />
939<br />
<br />
paper RAID: High Performance, Reliable Secondary Storage by Peter M. Chen et al.<br />
<br />
27.6.2 The optimal RAID level Disregarding any cost constraint, the best choice is almost always RAID 10. The issue is to understand for which activities the delivered RAID10 performance increase is worth the higher cost: In a data warehouse, the typical access pattern to data files is almost 100% reading; therefore the better performance delivered by RAID 10 arrays in write operations is irrelevant and the choice of a RAID 5 level is acceptable. On the other hand, in a typical OLTP environment, the huge number of write operations makes RAID 10 the best level for data files arrays. Of course, even for the OLTP data files, RAID 5 might be acceptable in case of low concurrency (see below for an explanation of the concurrency concept). The following is a brief analysis of pros and cons of each RAID level as regards Oracle RDBMS, based on V. Millsap’s Configuring Oracle Server for VLDB and on RAID: High Performance, Reliable Secondary Storage by Peter M. Chen et al.<br />
<br />
Data protection The impact of the RAID level on the I/O subsystem overall reliability has been subjected to many studies in the last fifteen years starting with David A. Patterson’ paper A Case of Redundant Arrays of Inexpensive Disks (RAID). We provide below only a few basic considerations, you can find a thorough analysis of RAID arrays reliability in G. A. Gibson’s paper Design of Disk Arrays for High Data Reliability and in Peter M. Chen’s paper RAID: High Performance, Reliable Secondary Storage. By means of an analysis based on Markov Chains, G. A. Gibson and David A. Patterson have clearly showed that (the reader is cautioned that computer scientists consider RAID 10 as a particular form of RAID 1): RAID 1 is generally speaking more reliable than RAID 5. RAID 5 with hot spare disks is generally speaking more reliable than RAID 1 without hot spare disks. RAID 1 with hot spare disks is more reliable than RAID 5 with hot spare disks. Optimal reliability for RAID 5 arrays is achieved with two hot spare disks. The reliability of a RAID 5 array with two hot spare disks is so high that the passage to RAID 1 with hot spare disks does not deliver a significant increase of reliability, thereby making the choice of the RAID level independent from the reliability issue.<br />
<br />
940<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
This very last consideration is a crucial lesson for the configuration designer:<br />
<br />
provided that the designer takes care to add a couple of hot spare disks, the choice of the RAID level can be based only on the cost and performance requirements. In addition to the RAID level, a few other parameters affect the I/O subsystem reliability. To make the discussion as simple as possible, we ignore here the impact of the introduction of hot spare disks. According to G. A. Gibson’s paper Design of Disk Arrays for High Data Reliability, assuming an independent failure model, the mean time to data loss (MTTDL) for a set of arrays is given by the following formula: MTTF 2 MTTDL = --------------------------------------------GN ( N + 1 )MTTR<br />
<br />
In this formula, MTTF is the hard disk mean time to failure, MTTR is the mean time to repair, and G is the number of error correcting groups each having N+1 disks. To clarify the meaning of these symbols, we point out that if the system has an hot spare disk, the MTTR is the time required to recover the array (error correcting group), while in this case it is the time to get a working disk, replace the failed disk and regenerate the array. The above formula clearly shows the risk of creating very large RAID 5 arrays. For example, for 10 (G=10) 10-disk (N=9) arrays (total 100 disks), the product GN(N+1) is 900, while for 5 (G=5) 20-disk (N=19) arrays (total 100 disks), the product GN(N+1) is 1900, thereby almost halving the MTTDL.<br />
<br />
Database files As regards write operations, RAID 10 arrays are much better than RAID 5 arrays, while for read operations the difference between the two levels is minimal. From this simple rule stem the following recommendations: Online redo log files: RAID 1 is strongly recommended. In case of very high performance requirements RAID 10 might be necessary. However, RAID 10 delivers performance benefits only in case of quite small stripe unit size. Archive redo log files: RAID 1 is recommended. However, the archive redo logs are not as critical for performance as redo log files. Accordingly, it is better to have archive redo logs on RAID 5 arrays than to have redo logs online redo logs on RAID 5 arrays. Temporary segments: RAID 1 or, even better, RAID 10, are recommended in case of many sort operations like is typical for instance in data warehouses. Data files: RAID 5 is acceptable for data warehouses, because the typical access pattern is reading and for small databases. Generally speaking, RAID 10 is the recommended RAID level.<br />
<br />
Chapter 27. Oracle<br />
<br />
941<br />
<br />
27.6.3 The optimal stripe unit size Tuning is a trade-off between performance and other operational requirements, particularly dependability, but it is also a trade-off between different and opposite performance requirements. For instance, if an array is used both for the typical OLTP activity, updates on small tables in a normalized DB, and for large reports generation, we have to decide the amount of performance degradation we are willing to accept on one activity in order to improve the performance of the other activity. The selection of the optimal stripe unit size is a trade-off between large I/O operations and small I/O operations. The target is to keep all disks working. If the concurrency, that is, the number of simultaneous logical I/O operations, is high, the stripe unit size should be large in order to be sure that each logical I/O is serviced exactly by one disk. On the other hand, if the concurrency is low, the stripe unit size should be small in order to be sure that each request is serviced by all the disks, thereby delivering a maximum throughput. A second parameter affecting the selection of the stripe unit size is the average I/O size. In abstract terms, the I/O size and the stripe unit size should be identical. However, because Oracle block boundaries are not necessarily aligned with stripe units, in order to increase the probability of matching a logical I/O operation with exactly one physical I/O operation, it is recommended to make the stripe unit size at least two times the average I/O size. The very first issue in designing the tuning an Oracle I/O subsystem is then to understand the concurrency level and average I/O size. Cary V. Millsap, in Configuring Oracle Server for VLDB, summarizes the sizing recommendations as described in Table 27-1. Table 27-1 Optimal stripe size<br />
<br />
Concurrency<br />
<br />
I/O size<br />
<br />
Recommended stripe unit size<br />
<br />
Affected arrays<br />
<br />
Low<br />
<br />
Small<br />
<br />
k times the db_block_size with k=2, 3, 4, etc.<br />
<br />
Low<br />
<br />
Large<br />
<br />
k times the db_block_size with k=0.25, 0.5, 1, etc.<br />
<br />
Log files<br />
<br />
High<br />
<br />
Small<br />
<br />
k times the db_block_size with k=2, 3, etc.<br />
<br />
OLTP data files (update activity)<br />
<br />
High<br />
<br />
Large<br />
<br />
k times the db_block_size times db_file_multiblock_read_count with k=2, 3, 4, etc.<br />
<br />
OLTP data files (reporting activity)<br />
<br />
Because the stripe unit size can have a significant impact on I/O performance, it is recommended for performance-sensitive systems to use technologies allowing<br />
<br />
942<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
the online change of stripe unite size. As regards IBM storage performance, this is one of the major reasons to use Fibre Channel storage (IBM TotalStorage DS4000 series) instead of SCSI storage (ServeRAID adapters). However, the reader is cautioned that besides changing the stripe unit size, it is often necessary to change the DB block size to improve performance. It is not at present possible to change the DB block size after DB creation. A new DB has to be created and data has to be moved from the old DB to the new DB.<br />
<br />
27.6.4 Oracle database block size The Oracle DB administrator is often asked to set the DB block size (DB_BLOCK_SIZE). This parameter is the one parameter that most affects I/O performance. In reality, the DB administrator’s experience with I/O access patterns might be minimal. The only way to change this parameter after the database has been created is to create a new DB and then to move data to that new DB. At installation time, little, if any, information is available on typical I/O size, and the DB administrator can use the following basic rules of thumb (see Cary Millsap’s paper Configuring Oracle Server for VLDB): The typical DB block size for OLTP systems is 8 KB or 16 KB The typical DB block size for DW systems is 16 KB or 32 KB; sometimes even 64 KB might be recommended A 4 KB DB block size can improve performance only in case of VLDBs consisting of thousands of very small segments. Moreover, to avoid unnecessary I/O operations, the block size should be a multiple of the operating system basic block size (allocation unit). As of release 9i, Oracle supports multiple block size in the same DB. This capability improves I/O performance, because it is possible to select the best block size for each tablespace in a database.<br />
<br />
27.6.5 Tablespace distribution As for tablespaces, the basic guidelines in tablespace design to achieve a good trade-off between the opposite requirements of performance and reliability are: Spread data into tablespaces so that each tablespace segment includes data with, as much as possible, similar characteristics, such as: – – – –<br />
<br />
I/O concurrency I/O size Access pattern (read versus write, sequential versus random) Time life span<br />
<br />
Chapter 27. Oracle<br />
<br />
943<br />
<br />
This data spread allows you to easily select the best stripe size unit, to get the most of the I/O storage subsystem and minimize fragmentation. Distribute data in tablespaces keeping in mind tablespace maintenance requirements, particularly activities requiring the tablespaces to be taken offline. For instance, because the system tablespace cannot be taken offline, it is advisable to avoid putting data in it as much as possible. Given these guidelines, the following is the list of most important recommendations. See Cary V. Millsap’s paper Configuring Oracle Server for VLDB for further information. Put only dictionary segments and system rollback segments in the system tablespace. This is both for performance and maintenance reasons. Put temporary segments in dedicated tablespaces. Put rollback segments in dedicated tablespaces. Do not put segments with short lifetime, causing high fragmentation, in tablespaces that are also hosting long lifetime segments. Put read-only segments in dedicated tablespaces. Put similar size segments in dedicated tablespaces. You can use the following query from J. S. Couchman’s Oracle 8 Certified Professional. DBA Certification Exam guide to verify which tablespace is a hot spot (frequently accessed for either reading or writing): select d.name, a.phyrds, a.phywrts, a.avgiotim from v$datafile d, v$filestat a where a.file# = d.file#;<br />
<br />
944<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
27.6.6 Example configurations Figure 27-2 shows a minimal configuration. RAID-1 was selected for sequential access (online and archived redo logs) drives, while RAID-5 was selected for random access drives. Online redo logs<br />
<br />
RAID-1<br />
<br />
Archived redo logs<br />
<br />
RAID-1<br />
<br />
Index<br />
<br />
Data<br />
<br />
RAID-5<br />
<br />
RAID-5<br />
<br />
Figure 27-2 Minimal example configuration<br />
<br />
Figure 27-3 shows a better, as well as more expensive, configuration. Online redo logs<br />
<br />
Archived redo Rollback logs segments<br />
<br />
RAID-1 RAID-1 RAID-1<br />
<br />
Index<br />
<br />
Data<br />
<br />
RAID-10<br />
<br />
RAID-10<br />
<br />
Figure 27-3 Better performing configuration<br />
<br />
Chapter 27. Oracle<br />
<br />
945<br />
<br />
Figure 27-4 shows a high performance configuration. Online redo logs Rollback Group A Group B segments<br />
<br />
Archived redo logs<br />
<br />
Temporary segments<br />
<br />
Index<br />
<br />
RAID-1 RAID-1 RAID-1<br />
<br />
RAID-10<br />
<br />
RAID-1<br />
<br />
RAID-10<br />
<br />
Data<br />
<br />
RAID-10<br />
<br />
Figure 27-4 High performance configuration<br />
<br />
Performance can still be improved, if necessary, as follows: Use RAID-10 for temporary segments if many large sorts are performed Use dedicated drives for hot tablespaces<br />
<br />
27.7 Monitoring DBMS performance In this chapter, we analyze DBMS performance through queries onto V$ views. However, this use of these views is only one of the tools that Oracle provides to monitor system performance. To monitor DBMS performance, Oracle provides the following tools: Oracle Tuning Pack The Oracle Tuning Pack is a set of six applications helping in analyzing system performance and identifying bottlenecks: – – – – – –<br />
<br />
Oracle SQL Analyze Oracle Expert Oracle SQL Tuning Wizard Oracle Index Tuning Wizard Reorg Wizard Tablespace Map<br />
<br />
You might find a thorough description of these applications, as well as how to use these application for system tuning, in the Oracle document Oracle Enterprise Manager. Database Tuning with the Oracle Tuning Pack.<br />
<br />
946<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Statspack Statspack is a command-line interface tool gathering performance relevant data. It is the successor of BSTAT/ESTAT tools and, as such, delivers much more functions. You can find more information about Statspack in Chapter 21 of Oracle 9i. Database Performance Guide and Reference. Dynamic Performance Views The term Database Performance Views identifies a large set of virtual tables collecting data on system performance. These views have names beginning with the V$ string. This is exactly the information source we have used all over the chapter to analyze system performance. A thorough list of dynamic performance views is provided in Chapter 24 of Oracle 9i. Database Performance Guide and Reference. Oracle Performance Monitor for Windows Oracle Performance Monitor for Windows 2000 and Windows Server 2003 is nothing more than Performance Monitor preloaded with Oracle 9i-related performance counters. In the following section we provide a little more information about this tool.The reader interested in an in-depth description of the tool should read Chapter 4 of Oracle 9i. Database Administrator’s Guide.<br />
<br />
27.8 Oracle Performance Monitor for Windows 2000 Chapter 19, “Analyzing bottlenecks for servers running Windows” on page 655 discusses the major subsystems in the server and lists the important counters to monitor using Performance Monitor. In that chapter, the following tables list the counters that you need to monitor to minimize performance bottlenecks in specific subsystems that are pertinent to Oracle databases: Processor counters: Table 19-1 on page 658 Memory counters: Table 19-2 on page 664 Disk counters: Table 19-3 on page 669<br />
<br />
Chapter 27. Oracle<br />
<br />
947<br />
<br />
Table 27-2 lists the values of Performance Monitor counters as recommended by Oracle for an Oracle database server. Table 27-2 Performance Monitor counters for Oracle<br />
<br />
Object: Counter<br />
<br />
Description and Recommended Values<br />
<br />
Oracle 9i Buffer Cache: %physreads/gets<br />
<br />
This counter should be as close to zero as possible. If the ratio is consistently over 20%, you should increase DB_BLOCK_BUFFERS to make the buffer cache larger. See also 27.5.2, “Database buffer cache” on page 933.<br />
<br />
Oracle 9i Data Dictionary Cache: %getmisses/gets<br />
<br />
This counter should be less than 10% or 15%. If the counter consistently increases above 15% while the applications are running it is recommended to increase the shared pool memory size by setting the SHARED_POOL_SIZE parameter.<br />
<br />
Oracle 9i Library Cache: %reloads/pins<br />
<br />
Reloads are an indication of the number of library cache misses on the execution step. This causes Oracle to implicitly reparse the statement and block. In addition, reloads can also occur if a buffer that includes an object definition is aged. Pins are reads or executions of the object in the library cache. Reloads should be kept to a minimum, as this causes library cache objects to be reinitialized and reloaded from disk. Reloads should be near 0 and this counter should not be more than 1%. If the value is higher than 1%, increase SHARED_POOL_SIZE.<br />
<br />
Oracle 9i Dynamic Space Management: %recursive calls/sec<br />
<br />
Non-zero counts on this value are caused by cache row misses or reads of segment headers. It should be as close to zero as possible. If it is high, consider increasing SHARED_POOL_SIZE.<br />
<br />
Oracle 9i Redo Log Buffer: redo log space requests<br />
<br />
This value indicates the number of times a user had to wait for space in the redo buffer. This should be near zero. High numbers indicate log buffers are too small and LOG_BUFFERS should be increased.<br />
<br />
Oracle 9i Sorts: sorts in memory/sec<br />
<br />
This value should be higher than the Sorts on Disk/Sec counter. If not, consider increasing SORT_AREA_SIZE.<br />
<br />
Oracle 9i Sorts: sorts on disk/sec<br />
<br />
This value should be lower than the Sorts in Memory/Sec counter. If not, consider increasing SORT_AREA_SIZE.<br />
<br />
Oracle 9i Data Files: phyreads/sec<br />
<br />
This counter can be used to reveal which data files are being read the most and whether they should be moved to other disks.<br />
<br />
Oracle 9i Data Files: phywrites/sec<br />
<br />
This counter shows which data files are being written to the most. If the data files of the temporary tablespace are being written heavily, this might indicate that sorts are being done to disk rather than in memory. If this is the case, consider increasing SORT_AREA_SIZE.<br />
<br />
948<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
27.9 Summary Oracle performance tuning can appear to be an intimidating task due to its many parameters. However, it is this flexibility to adapt to many application environments that makes Oracle such a robust and powerful database management system. With only a basic description of the architecture and some understanding of a few parameters, a user can increase the performance of an Oracle database in a short time. This chapter provided some guidelines and suggestions that can improve Oracle performance significantly over a default installation. It provided basic descriptions of tools that you can use to gain further insight into the inner workings of Oracle and, therefore, provide the user with a mechanism to measure and improve performance in all environments.<br />
<br />
Chapter 27. Oracle<br />
<br />
949<br />
<br />
950<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
28<br />
<br />
Chapter 28.<br />
<br />
Microsoft Windows Terminal Services and Citrix Presentation Server The terms thin-client computing or server-based computing have become popular catch-phrases in the IT industry in recent years. In many respects, technology has come full circle, with computing resources that were until recently broadly distributed to the user’s desktop now coming back to the data center. Industry professionals have come to realize the ongoing battle that supporting a distributed computing environment brings. Thin-client computing is often touted as the weapon to combat this never-ending fight. While not providing all the answers to these challenges, thin client, otherwise known as server-based computing, can make sizeable inroads. Windows Terminal Services1 by itself or combined with Citrix Presentation Server2 has become the industry standard for thin-client computing. This chapter examines these technologies and how to implement these products for best performance. Much of this chapter focuses on terminal server architecture, sizing and placement rather than pages of specific tuning techniques for either platform. In almost all instances, suggested performance tuning suggestions are 1 2<br />
<br />
Product screen captures and content reprinted with permission from Microsoft Corporation. Portions of the content in this chapter are reprinted by permission of Citrix Systems, Inc.<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
951<br />
<br />
complimentary to both Windows Terminal Server and Citrix Presentation Server platforms than being appropriate only for one or the other. In this chapter, we discuss the following topics: <br />
<br />
28.1, “Thin-client or server-based computing” on page 952 28.2, “Server sizing and scalability” on page 954 28.3, “Server architecture and placement” on page 958 28.4, “Microsoft Windows Terminal Services” on page 961 28.5, “Citrix Presentation Server 4.0” on page 970 28.6, “Common tuning options” on page 974 28.7, “Load balancing” on page 977 28.8, “Monitoring Windows Terminal Services” on page 979<br />
<br />
28.1 Thin-client or server-based computing IT management and technologists alike have long recognized the significant challenges that managing distributed computing resources creates. The cycle of maintaining hardware, firmware, operating systems, software, application updates, virus signatures and security patches on “fat” personal computers is unending. This, combined with business driven change processes and the sizeable overhead of supporting, securing and administering what can be tens of thousands of user desktops and laptops across hundreds of locations, makes managing distributed computing resources slow, time-consuming and painfully expensive. Enter thin-client or server-based computing. By bringing as many components of the distributed computing environment as possible back to the data center, and thus physically and logically closer to the architects, system engineers and support personnel who design and maintain these systems, considerable cost and time savings can be realized. Through the use of server-based computing, administrators can now update and change systems in one central location, often on a single system. Changes that might have taken weeks or months to update or roll out to thousands of distributed computer, can now be performed on centralized computing resources in minutes or hours. By consolidating distributed applications into centralized server farms, higher levels of fault tolerance and systems management can be offered than previously. Using central technology servers in central data centers are far easier to backup, monitor, and maintain than a field of widely dispersed computing systems. Centralized servers are closer to other computing resources they might need to access and can be implemented with load-balancing technologies to deliver higher levels of system uptime and availability.<br />
<br />
952<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
A typical thin-client computing session is comprised of three fundamental elements: The client computer that requires access to shared or common resources. This client might be running a platform including Microsoft Windows, Apple Macintosh, UNIX, Linux or even a number of hand-held computing devices. A centralized server that permits multiple users to work interactively with the resources on the server. These resources (typically applications) are often presented in such a way that the user experiences them as though there were locally running. The thin-client protocol that connects the client computer to the central server. Server-based computing is far more than a “remote-control” session on a central server. These central servers run many concurrent interactive sessions, each running on the same host but with each running seemingly independent of each other. The central server operating system must be designed to handle these many simultaneous sessions, protecting each session for each other and keeping the operating system kernel protected. The Terminal Services component of Microsoft Windows Server delivers on this requirement and Citrix Presentation Server extends its reach and functionality. High levels of infrastructure performance and throughput are critical in a thin client computing environment. In many instances, thin client technology is used to deploy and replace applications that typically are installed locally on a client desktop. This means that the server performance and network bandwidth needs to operate at a sufficient level to ensure that the user feels as though the remote application is performing like a local installation would. All the administrative benefits are lost if the user-experience is considerably more painful than it was to be running the application locally. Previously, with distributed computing, users have each loaded applications individually on their own local desktop or mobile computers and consumed local computing resources. With server-based computing, the load is placed centrally back on the servers offering the applications. Typically, with every user connecting to the server and launching an application, the server loads another instance of the application into memory, consuming server memory, CPU, and network resources. To realize the maximum scalability benefits of thin-client computing, a system engineer might choose to run as many sessions as possible onto as few servers as available. Thus, appropriate server sizing becomes critical. While offering many manageability benefits, thin client technology pushes the requirement for adequate processing power from the client back to the server. This means that the servers running WindowsTerminal Services and Citrix<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
953<br />
<br />
Presentation Server need to be configured with hardware that is able to handle the additional load that having many simultaneous desktop sessions will impose. Take for example, a terminal server running Microsoft Word for 30 users. Unlike a desktop client that needs to run only one instance of Word, a Terminal server must have sufficient resources to run 30 copies of Word simultaneously. Administrators and architects must be conscious of the impact that loading centralized client applications will have on servers—either as stand-alone servers or in one of many servers in farms. The load that the application processing itself will place on the server must be considered in addition to the load that the many clients loading multiple instances of the application will create. Well architected server-based computing designs typically do not use the local or directly attached storage of the terminal servers themselves, but instead take advantage of nearby file servers in the same data center. This configuration takes the burden for having large amounts storage space off the local terminal server. Thus the most heavily impacted server subsystems are typically, in order, memory, processing power and to a lesser extent, network.<br />
<br />
28.2 Server sizing and scalability Determining the appropriate hardware for a server based computing environment introduces many design considerations that are not normally required with typical client-server computing. This section aims to address some of these.<br />
<br />
28.2.1 Scale up or scale out When designing a terminal server solution to host many users, the issue of implementing fewer, larger servers (scaling up), versus, a larger number of smaller specification servers (scaling out) is normally considered. Hosting your terminal services function on fewer larger servers decreases software licensing and the management and operational overhead associated with additional hardware. It also reduces the additional administration that comes with an another operating system instance. IBM System x servers such as the x3850 are well suited to such a scale-up solution, with it’s four processor sockets, 64 GB memory capacity, and it’s many high availability and fault-tolerance features. It is important to realize that a single large system can create fewer points of failure so ensure the correct server is selected for the job to maximize uptime.<br />
<br />
954<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Scaling out In a load-balanced terminal services scenario, a failure of one of only two large specifications servers will immediately remove half of the server processing power available to clients. By contrast, a failure of one of a smaller server, when running as a member of a larger farm of servers, will likely have less of an overall impact. What must be said however is that larger systems tend to have much greater resiliency by design, with more redundant components, so these higher-end servers are less likely to suffer a complete failure. Loading fewer, larger servers with many applications rather than scaling out to a larger number of smaller servers can also greatly increase the complexity of the installation. Applications need to be socialized with one another to confirm they can coexist with issue, which is not always a straightforward process to test. Software vendors will also often not support their application if it is running on a server with other applications. This presents an ongoing source of frustration of server administrators but it is a reality that must be considered in any design. Similarly, performing server changes or administration to large, heavily laden servers can be difficult. Finding an outage window that suits everyone using a given server can be difficult if it is supporting multiple, disparate applications. Scheduled maintenance to servers is typically easier to carry out if the impact of taking a single system down is less. Server hardware today is rapidly becoming smaller in physical size. Dual-processor servers are easily available now in one rack unit (1U) forms or in even more compact blade server design. These smaller servers still provide the memory and processing capabilities required to support central terminal server requirements without the need for large internal disk storage that more conventional rack mounted servers offer. This makes them ideal candidates for centralized Windows Terminal Server and Citrix Presentation Server farms. Bolting on additional capacity with another smaller server is certainly a more cost-effective and flexible approach than adding a much larger and expensive single additional larger server also. The financial break-point to justify another server when using fewer, larger systems is considerably higher than that required to substantiate another smaller system when growth demands it.<br />
<br />
Scaling Up Windows Server 2003 and Citrix Presentation Server 4 yield much higher scalability than previous versions meaning physical hardware can be utilized much more efficiently. This is especially the case now with the introduction of x64 editions of Windows Server 2003. With previous versions of Windows, adding additional processors in a thin-client server environment did not offer linear increases in performance or scalability in terms of support user connections. With an x64 environment now, kernel memory constraints are no longer a<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
955<br />
<br />
restriction to scaling up individual servers. As as result, whilst there can be notable management and administration challenges with running fewer, larger servers, hardware scalability is no longer the limitation it was in the past. Server consolidation to fewer larger machines is a popular concept today, especially in light of server virtualization technologies. Virtualization may not however, be appropriate when designing thin-client solutions. This is especially the case where a server is typically only running one or a few applications for terminal server clients. Providing the applications can scale as the operating system scales upwards, there may be little gained by running multiple operating system instances on a single physical server through virtualization. The additional virtualization layer will likely add unnecessary overhead and complexity. It would be far better to simply scale a single operating system instance on the physical server as far as it can grow. Each environment will need to weigh the pros and cons of scaling up the terminal services infrastructure occasionally with fewer, larger servers compared to the greater server growth seen in environments that choose to run a larger number of smaller specification servers.<br />
<br />
28.2.2 Sizing considerations Given the considerably higher CPU and memory load that a centralized Terminal Server typically is used for over that of a more conventional server purpose, it is important to size the server hardware specifications to suit requirements. Many users share the server environment, so it makes sense to purchase enterprise class hardware that is fully redundant, scalable, and easily user-serviceable, including hot-swappable redundant drives and power supplies. With server hardware becoming practically commodity items today, and processor and memory prices decreasing every day, it is not sensible to save a few dollars here or there when this proportionately small amount of extra expense might mean a good number more simultaneous user sessions on the same server. This is not to say that a new terminal server should be ordered with the maximum amount of memory and CPU power that the machine can possibly accommodate; appropriate analysis of the system is required as described below. It does however mean that if for a few another few hundred dollars another gigabyte of memory can be easily added to the server, or a dual-core processor can be used instead of single-core, it will be easier to justify and add this at the initial time of server purchase than down the track when the server is busy and harder to get an outage window and spare parts are considerably more costly. It can also mean the usable life of the server is extended a little longer which can assist with financial considerations such as depreciation.<br />
<br />
956<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Consider the following factors when determining the right hardware specification for servers that run Windows Terminal Services or Citrix Presentation Server: The operating system itself Microsoft suggests Windows Server 2003 has a minimum recommendation of 256 MB. We would recommend at least doubling this to 512 MB as a minimum and, ideally, to 1 GB of RAM if combined with Citrix Presentation Server. Concurrent user application sessions If the application you will be loading on a given server requires 50 MB of memory and you plan to support 60 concurrent user sessions, you need an additional 3 GB of memory. Note: The key here is the maximum number of concurrent sessions the server can support, not the total number of users the server will be supporting over time. Processor requirements Processor speed is largely driven by the what is available in the market at any given time so there is little option for consideration here. In addition, calculating actual CPU cycles that are required for a given number of application instances is not easy. The number of processors and number of cores per processor however is able to be chosen. Our recommendation is to purchase one or two dual-core processors for your system depending on your budget. If you suspect you might require more processing power, the decision will come down to whether you want to scale up the same server or expand out to another server. Having a server that can scale up to four or more processors is more convenient but is an expensive overhead to carry if it is never used. If the opposite is true and the dual-core dual-CPU recommendation is overkill for your given requirement, it is not a considerable waste of expense. Nature of the users and the application type “Standard” users do not place demands on the processing and memory subsystems of your server the same way “power” users do. A given server might be quite capable of supporting dozens of concurrent users who only access basic word processing functions periodically throughout the day. The exact same server, however, responds very differently where a smaller number of very active users are performing intensive mathematical spreadsheet functions. Local application processing<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
957<br />
<br />
If the terminal server is going to be performing any local application processing itself as part of client/server model the load this might place on the server needs careful consideration. We would typically recommend that the presentation layer (that is, Terminal Services or Citrix Presentation Server) is kept separate from any processing or data store requirements of a typical n-tier server architecture. Ideally, a pilot environment should be established with users carrying out typical workloads on the server while load monitoring takes place. Some applications do not scale well on a shared server, and it is better to determine this in a pilot rather than after the system has been launched into production with budgets finalized.<br />
<br />
28.3 Server architecture and placement Improving performance in a thin-client server design is often more about the overall system architecture and the placement of the servers relative to one another and the users accessing them than it is about sizing server hardware appropriately. Even with all the factors that we have mentioned in preceding chapters taken into consideration, a correctly sized terminal server will still offer a poor user experience if the applications it runs require connectivity to back-end servers that are in a data center on the other side of a over-subscribed global WAN link. Careful consideration must be made as to the placement of the terminal server infrastructure. Whether to place the server closer to the users and thereby optimizing their experience from a presentation perspective or whether to locate the server in the data center nearest the other services it needs to communicate with can often be a careful balancing act. By its very nature, thin-client technology lends itself to putting terminal servers in a centralized data center with clients accessing the systems through thin remote communications protocols. This topology of course assumes that all given “hops” across network links have sufficient bandwidth and are not affected adversely by the latency issues that are experienced with very long wide area network links—typically those spanning entire countries or between continents. Thin-client protocols, such as RDP and ICA, are optimized frequently. However, if network performance cannot be guaranteed, then in some instances, it might make sense to place the terminal server closer to the user, in a data center that is closer logically (less hops) or geographically (reduced latency). Managing the type and time of data using a given network link combined with quality-of-service (QoS) and packet-shaping technology can aid greatly in improving network<br />
<br />
958<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
constraints that might otherwise prevent a terminal server from being located in the most sensible location. If inter-server communications between terminal servers and back-end servers is considerable, it is ideal to locate these servers as close to one another as possible. Consider the location of the following servers to which a terminal server might need access and the amount of traffic transferred between the two when determining correct server placement: Authentication servers such as Active Directory domain controllers Public Key Infrastructure servers such as certificate authorities, issuing certificate servers and associated certificate revocation lists (CRLs) Naming services such as domain naming services (DNS) or LDAP Terminal Server licensing servers File servers and print servers Application processing servers Database servers Other terminal servers including other presentation/terminal servers and other central service functions such as Citrix data stores and zone data collectors Another consideration is that any service that the terminal server simply cannot function without, should an intermediary WAN link go down, must reside on the same local network as the terminal server itself. This recommendation might require additional servers to support these services or might impact where you locate the terminal servers in your design. Factoring in this consideration ensures application availability. Critical services that a typical Windows Terminal Server cannot function without likely include authentication and license servers and quite possibly naming services servers such as DNS.<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
959<br />
<br />
28.3.1 User profiles Administrators often like to offer users a very similar desktop experience regardless of whether the user is working locally on their own desk machine or on any number of remote terminal server desktops. The use of Windows roaming profiles permits this experience by storing the users desktop settings, documents, start menu, and other user-specific components on central servers and then copying the profile from and to this central location at logon and logoff time. Distributed environments that use Windows roaming profiles encounter considerable challenges when determining the best place to store the files that make up a user profile. As user profiles can become quite large, the logon and logoff experience can be painfully slow if the user profile is stored in a remote location. This situation becomes particularly problematic when the user works with both thin-client applications served on remote servers and their normal local desktop session, both of which might require access to the same roaming profile. Making the problem even worse again is when the user needs to access terminal servers that are located in distributed locations, as the common user profile still only typically resides in one central location. This all too common scenario can be managed through any or a combination of the following mechanisms: Using mandatory profiles. These profiles are locked-down, optimized and generally load faster as they do not change or grow in size over time however they allow no individual user personalization. Do not use roaming profiles. This option typically means the user interface experience will not be the same from server to server but the performance improvements might outweigh the user advantages of a consistent desktop. Limiting roaming profile size. This mechanism can be achieved through folder re-direction or excluding specific directories or specifying a maximum profile roaming profile size. You can find more information about these techniques using the following links: http://support.microsoft.com/kb/232692 http://support.microsoft.com/kb/188692 http://support.microsoft.com/kb/888095 http://support.microsoft.com/kb/290324 http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/ regentry/93591.mspx?mfr=true<br />
<br />
960<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Specify multiple roaming profiles. Group policy allows an administrator to specify that given terminal servers should copy a user’s roaming profile from a different location than other terminal servers. This mechanism is a particularly helpful feature in managing profile issues. See the following link for more detail: http://technet2.microsoft.com/WindowsServer/en/library/ae1fbb95-eb16 -4ea6-ba8c-6ca7d60b5f181033.mspx?mfr=true Do not retain locally cached profiles. Having the user profile stored local in any sized terminal server farm offers little advantage and consumes disk space unnecessarily. See the following links for more information: http://support.microsoft.com/kb/173870 http://support.microsoft.com/kb/196284 http://support.microsoft.com/kb/214653 Exclude the use of roaming profiles on selected Terminal Servers. This feature, which was introduced in Windows Server 2003, allows you to enable or disable roaming profile functionality on selected servers. See the following links for more detail: http://support.microsoft.com/kb/817361 http://www.microsoft.com/technet/desktopdeployment/articles/102505ev .mspx<br />
<br />
28.4 Microsoft Windows Terminal Services Microsoft Windows Server 2003 Terminal Services provides remote access to a server desktop by means of thin client software, operating as a terminal emulator. This client is referred to as the Remote Desktop Connection or Terminal Services Client. It is native to Windows Server 2003 and Windows XP and can easily be added to Windows 2000 and Windows NT client. The thin protocol supporting the Remote Desktop Connection or Terminal Services client connectivity is the Remote Desktop Protocol, or RDP. RDP is a highly optimized protocol to ensure only the most essential traffic is transmitted over the network. Terminal Services sessions transmit only the user interface of a program to the client. The client then returns keyboard and mouse strokes and clicks to the server for processing.<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
961<br />
<br />
28.4.1 Remote Administration versus Application Mode Terminal Services can be deployed on the server in either application server mode or remote administration mode. When used in remote administration mode, which is installed and operational by default in Windows Server 2003, Terminal Services can provide a means to remotely control your server from virtually anywhere on a connected network. The Windows 2000 Terminal Services Remote Administration Mode is now called Remote Desktop for Administration in Windows Server 2003 and supports the Remote Desktop Protocol (RDP) 5.1 feature set. In application server mode, Terminal Services provides an easy way to distribute Windows programs using a network server. Application server mode has additional licensing requirements for both the clients and server. It is Application Mode that allow multiple users to connect and run desktop and application sessions on the central server, and thus the focus of this chapter. Terminal Services in Application mode is installed through the Add/Remove Windows Components of the Add or Remove Programs Control Panel in Windows Server 2003. During installation, two security options are available: Full Security or Relaxed Security. The former modifies permissions on the registry and file system to defaults that Microsoft believes are typical for most environments. It does however cause known issues with some applications. By using Relaxed Security, less compatible applications are more likely to run trouble-free while still not compromising the full security of the server. We recommend the use of Full Security, unless an application requires the looser permissions that the Relaxed Security option provides.<br />
<br />
28.4.2 Remote Desktop Sessions The fundamental concept of accessing an application on a server that is running Windows Terminal Services is that it permits a user to log on to a server, start a remote desktop session on that server, and then launch the application. So in effect, for a typical Windows client, the user is running a Windows desktop within their own desktop environment. These remote desktop sessions and applications can be tailored considerably to make them appear very “local” to the user.<br />
<br />
962<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Windows Terminal Services does permit the administrator or the client to control the application that a user will access when connecting to a given server through the use of initial programs. This program launches when the user connects through an RDP session to the server desktop, so they see the application only, not the server desktop session itself. This setting is made at either the client or in the RDP connection specific settings in the Terminal Services Configuration MMC, as shown in Figure 28-1.<br />
<br />
Figure 28-1 Initial programs<br />
<br />
Citrix Presentation Server extends this functionality into seamless published applications. These applications have the same appearance as an application that has been launched locally, with windows that can be resized. From a performance perspective, there is little difference between a remote desktop session directly on the server and launching an initial program or seamless window as the server load is identical. Administrators might choose to use initial programs or Citrix seamless applications to offer a tighter level of security. However, most of the advantages are for an enhanced user experience.<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
963<br />
<br />
28.4.3 Tuning Windows Terminal Server Many of the recommendation performance tuning methods included here are similar to those that we list in Chapter 11, “Microsoft Windows Server” on page 295. Reference this chapter for more information of the performance tweaks that we list in this section and many more that can yield further improvements for you. Particularly relevant to terminal servers are the sections on processor scheduling, virtual memory and page file settings, the file system cache, and TCP/IP tuning recommendations. You can set most of the options that we discuss in this section through the Terminal Services Configuration MMC that is installed by default on all Windows 2003 Servers.<br />
<br />
Applications It is recommended that you disable extra features within applications that are not required. An example of these are the autocorrect options with Microsoft Word and the Office Assistant in Microsoft Office. By disabling these features within the user applications, you reduce the load on the server. You can make these features available to individual users as required, but simply disable them by default. The objective is to only enable the features that are required to avoid tying up system resources unnecessarily.<br />
<br />
964<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
RDP Security and Encryption You can adjust the level of client-server security and encryption that is used for communications between the server and the client. The lower the security and encryption level, the fewer resources this feature consumes. If the data that is sent between the server and client is not confidential, you should consider a lower level of encryption to improve client performance and to reduce the burden on the server (see Figure 28-2).<br />
<br />
Figure 28-2 RDP Encryption levels<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
965<br />
<br />
RDP Client session time-outs By default, all disconnected Terminal Server sessions are retained upon disconnect. The length of time that the session is retained can be set from 1 minute to never (thus disabling the timer). When the disconnected session times out, the server resets it, which frees system resources for additional settings. Figure 28-3 shows where you set the timeout values.<br />
<br />
Figure 28-3 Session settings<br />
<br />
Thin client sessions can be restricted by use of the Active Session Limit setting, which determines how long the session can be active on the Terminal Server until it is disconnected automatically. This feature is valuable when a user accidentally leaves a session open or if you want to restrict usage due to limit impact on server or network resources. You can also adjust the Maximum Idle Time (denoted as time without connection activity) that is allowed before a session is reset. You can adjust this setting through the Idle Session Limit parameter. Again, this setting allows resources to be freed by disconnecting users who have been idle for any length of time (from 1 minute to never, disabling the timer).<br />
<br />
966<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
RDP Network card affinity Windows Server 2003 Terminal Services has the option to allow the clients to attach to the server through all or one of the network cards installed on the server. If the server is configured to allow clients to connect to one of the network cards, this configuration frees the remaining network cards to service the network communications with other possible back-end servers (for example, a back-end file, application or database server), as shown in Figure 28-4.<br />
<br />
Figure 28-4 RDP network adapter<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
967<br />
<br />
RDP Client resource settings Windows Server 2003 Terminal Services lets you disable the client virtual channels. Each feature that is disabled frees server resources to be available for other purposes and reduces the size of the RDP connection. We highly recommend disabling all the RDP client settings, as shown in Figure 28-5.<br />
<br />
Figure 28-5 RDP Client settings<br />
<br />
968<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Terminal Services Server settings The settings that we have discussed thus far all relate to RDP connection specific settings. The Terminal Services Server settings encompass all connection settings for the server. You configure these settings also in the Terminal Services Configuration MMC. We recommend the values listed in Table 28-1 for performance related settings for optimal terminal server performance. (See also Figure 28-6.) Table 28-1 Terminal Services Server settings<br />
<br />
Server setting<br />
<br />
Attribute<br />
<br />
Delete temporary folders on exit<br />
<br />
Yes<br />
<br />
Use temporary folders per session<br />
<br />
Yes<br />
<br />
Active Desktop®<br />
<br />
Disable<br />
<br />
Permission compatibility<br />
<br />
Full security<br />
<br />
Restrict each user to one session<br />
<br />
Yes<br />
<br />
Figure 28-6 Terminal Services Server settings<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
969<br />
<br />
28.5 Citrix Presentation Server 4.0 Citrix Presentation Server builds on Windows Server 2003 to deliver a more feature rich, flexible and compatible terminal server experience than the native product delivers. At time of writing, the current version of Citrix Presentation Server available is 4.0. Presentation Server offers considerable improvements in performance, mobility, management, application configuration, security, scalability and client compatibility than previous versions of Presentation Server (also known as Citrix MetaFrame) have. While still the flagship offering of the Citrix product range, Presentation Server is now often delivered as part of an impressive well-integrated suite of products that deliver sophisticated thin-client, Web application, collaboration, security and remote access solutions. These combined products form what Citrix calls its Application Delivery Infrastructure. While the Citrix product suite is extensive, the focus of this section is primarily on Presentation Server 4.0 and its interaction with the underlying Windows Server 2003 operating system. Building upon the native Windows Server 2003 Terminal Server functionality, it is critically important to ensure the underlying operating system is optimized for performance. Following the recommendations in Chapter 11, “Microsoft Windows Server” on page 295 on Windows Server tuning will go a long way towards delivering the ultimate thin-client experience.<br />
<br />
28.5.1 Presentation Server and Windows Terminal Services Citrix Presentation Server is an extension to Windows Server 2003 Terminal Services. Presentation Server provides essentially the same function set to clients as Terminal Services does, but Presentation Server broadens the client base and manageability of those clients. This software is installed on top of Windows Server 2003 Terminal Services (Application mode) to deliver a much more sophisticated user experience to a broader range of clients such as Java, Linux, UNIX, OS/2®, Macintosh, and other client devices. The standard Remote Desktop Client (RDC) can connect to the server using only the TCP/IP protocol. Presentation Server adds the ability for clients to connect using protocols not supported by Terminal Services, such as IPX™, NetBIOS, and Async (modem) using the Independent Computing Architecture (ICA) protocol. Citrix documentation suggests the ICA protocol is more highly optimized than native RDP to Terminal Services. A single ICA session requires<br />
<br />
970<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
at most only 20K of bandwidth, and this will be reduced again depending on the applications used. Presentation Server also provides clients with services such as published applications. By publishing an application on the server, an administrator makes the application available to users as a resource. Users of client machines can select an application and launch it for use. Unlike standard Windows Terminal Services, applications published through Presentation Server are not “wrapped” in a remote server desktop session; instead, they look and behave like a local client desktop application, improving the user experience. Users with Windows 95, 98, NT, 2000, and XP and Java clients can run these applications in seamless mode, which means that the remote application appears as though it is running locally. You can resize and move the application window around the desktop alongside other, perhaps local, applications.<br />
<br />
28.5.2 What’s new in Version 4.0 Version 4.0 of Presentation Server introduces many new features to the already impressive line-up offered by its predecessors Presentation Server 3.0 and MetaFrame XP. These new features include: Support for more applications: Through the use of new technologies such as application isolation environments and virtual IP addresses, applications can be deployed into thin-client Citrix environments with greater reliability and requiring less servers. Increased user density: New features such as CPU virtualization management and virtual memory optimization mean individual Presentation Servers can support more users. The recently available 64-bit version of Presentation Server running on the 64-bit version of Windows Server 2003 will provide considerably greater user density, by a factor of up to 3-times greater than was seen on the 32-bit equivalents. Printing improvements: A new universal printer driver greatly improves print device compatibility and makes printing up to four times faster through reduced memory and network bandwidth. Windows & UNIX together: The Enterprise Edition of Presentation Server combines the Windows and UNIX offering into one product, simplifying management and licensing.<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
971<br />
<br />
28.5.3 Performance features of Presentation Server 4.0 The following list of key performance and scalability related enhancements indicates where Presentation Server extends the functionality of native Windows Server 2003 Terminal Services. CPU Utilization Management: The CPU management feature of Presentation Server ensures that a processor intensive process initiated by one user does not impact or degrade the performance of other sessions. Virtual memory optimizations: DLL rebasing for applications is performed in order to reduce the amount of memory conflicts when loading DLLs, resulting in a reduction in overall memory requirements for some applications. and enabling a single server to support more concurrent users. Session sharing: When a user requests a second application on Terminal Services, the application is launched in the existing user session. This allows the application to start up almost instantly (as opposed to creating an entirely new session), also reducing memory and CPU consumption on the server. Enhanced Application and printing performance: Independent studies have shown that Presentation Server enhances standard Terminal Services application and printing performance by 2 to 3 times. Bandwidth Tuning: administrators can ensure optimal performance by limiting the amount of bandwidth used by activities like printing and file transfers; a particularly helpful feature for remote offices with constrained wide-area-network links. Support for high-latency WAN links: The SpeedScreen Latency Reduction feature of Presentation Server optimizes performance by echoing keystrokes and mouse clicks locally, reducing the impact of high-latency WAN links to the user experience. File transfer performance is also improved and not adversely affected by network latency issues. Reduced bandwidth consumption: Independent studies have shown that the ICA protocol is more network efficient than the native Terminal Services RDP protocol, by between 25 and 50 percent. Server-to-client URL detection: Running Web browser sessions across thin-client environments is not efficient and typically delivers a poor user experience. When clicking on Web URLs within an application running on a remote Presentation Server, the link is opened with the local client browser instead of launching the browser on the Terminal Server. Distribute load relative to sessions, CPU & memory: Citrix load-balancing establishes user sessions across a group of load-managed servers based on configurable parameters like session count, CPU utilizations, memory consumption, time of day, and so forth, balancing server workload and improving performance for both the server and the user experience.<br />
<br />
972<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
For more information, see the following links: http://www.citrix.com/English/ps2/products/feature.asp?contentID=23304 http://www.citrix.com/site/resources/dynamic/salesdocs/Citrix_Value_add _to_Windows_Terminal_Services_2003.pdf<br />
<br />
28.5.4 Tuning Citrix Presentation Server Citrix Presentation Server is natively highly optimized to deliver a very good quality thin-client experience to the user. It comes with many features that can be optionally implemented or tuned to improve the experience even further for a given environment. Several of these are described below.<br />
<br />
Disabling unnecessary features of Presentation Server When configuring and publishing applications in Presentation Server, ensure that you carefully select only the client-side options that are required. Support for sound, encryption and the window size and number of colors supported should be considered carefully to determine how useful these really are. You can minimize network traffic by disabling any unnecessary components including wallpaper, screen savers and printer, clipboard, audio, COM, and LPT mappings. Restricting the number of features that are available to clients offers performance benefits to the individual user session and reduces the number of settings that must be negotiated at session setup and then maintained throughout the session life.<br />
<br />
Enable SpeedScreen acceleration SpeedScreen is a Citrix application aimed at improving the user experience for published applications, particularly over slow WAN links. Speed Screen Acceleration comes with three options in Presentation Server 4.0: SpeedScreen Browser Acceleration: Improves the performance of published applications that use embedded images in HTML pages SpeedScreen Flash Acceleration: Improves the presentation of Flash animations of an ICA session SpeedScreen Multimedia Acceleration: Improves the performance of streaming audio or video to ICA clients. The SpeedScreen Acceleration options are controlled for applications through the Citrix Management Console. All three options are enabled by default.<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
973<br />
<br />
Clients need to run the ICA client version 7.0 or later to take advantage of the three SpeedScreen features. For more information about Speed Screen, see the following: http://support.citrix.com/article/CTX104735<br />
<br />
Enable the ICA keep-alive query The Citrix Management Console in Presentation Server 4.0 you can configure ICA keep-alive to query user sessions at regular intervals to ensure they have not been disconnected due to network link failure. If a given session does not respond to the Keep Alive query, Presentation Server 4.0 places the session in disconnect mode automatically so that the user can reconnect at a later time and not have to start a whole new session. We highly recommend the use of this setting to ensure only the minimum number of actual client sessions are running on a given server which will be best for session performance. To enable this setting, do the following: 1. Open the Citrix Management Console. 2. Right-click to selected the desired farm node. 3. Click Properties. 4. Click ICA Keep-Alive. 5. Select ICA Keep-Alive and set a time-out value of 60. 6. Click OK. While this setting works specifically with the ICA protocol only, it is very similar to the operating system setting for TCP/IP keep-alives that we describe later in 28.6.2, “TCP/IP keep alives” on page 976.<br />
<br />
28.6 Common tuning options The following optimizations build on those already detailed in Chapter 11, “Microsoft Windows Server” on page 295 on Microsoft Windows Server. Review that chapter and apply the optimizations listed there (where appropriate) in advance of the tuning suggestions that we list in this section. The tuning options that we discuss in this section are ones that add particular value to both standard Windows Terminal Services and Citrix Presentation Server environments.<br />
<br />
974<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
28.6.1 Remove Windows visual effects To optimize the remote desktop experience for users, it is recommended that you disable all visual effects that Windows permits unless those effects are absolutely necessary. While visually pleasing to some users on a standard Windows desktop session, it is not appropriate for shared environments where performance is critical. To disable visual effects, follow these steps: 1. On the Terminal Server console, right-click the desktop and choose Properties. 2. Select the Appearance tab. 3. Click Effects. 4. A window similar to that shown in Figure 28-7 displays. Deselect every check box for best performance as shown in the figure.<br />
<br />
Figure 28-7 Windows visual effects settings<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
975<br />
<br />
28.6.2 TCP/IP keep alives Networks that suffer from high congestion or from high latency can cause client sessions to drop out. When the user reconnects to the server, they are not reconnected automatically to their previous session because the server is not aware that the initial session was disconnected. Thus, the work is interrupted, and the user cannot gain access to the previous session. A prevention for this problem is to enable TCP/IP keep alives. With keep alives enable, the terminal server is aware of any dropped sessions sooner than it might normally be aware of them. It is worth noting that enabling keep alives might keep demand-dial WAN links up and running and, thus, might potentially increase the expense of such services. Set the keep alive parameters carefully to ensure that TCP/IP communications do not timeout earlier than expected. You might need to tune these values to prevent this type of behavior. To enable keep alives, you need to set two values in the terminal server registry. The KeepAliveTime value controls how often the server attempts to verify that an idle connection is still there through sending a keep alive query packet. Key: Value: Data type: Range: Default: Recommendation: Value exists by default:<br />
<br />
HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters KeepAliveTime REG_DWORD (time value in milliseconds) 0x1 to 0xFFFFFFFF 0x6DDD00 (7200000) (2 hours) 0x0000EA60 (60 seconds) No, needs to be added<br />
<br />
The KeepAliveInterval determines the interval that separates keep alive retransmissions (as determined by KeepAliveTime) until a response is received. Key: Value: Data type: Range: Default: Recommendation: Value exists by default:<br />
<br />
HKLM\SYSTEM \CurrentControlSet \Services\Tcpip\Parameters KeepAliveInterval REG_DWORD (time value in milliseconds) 0x1 to 0xFFFFFFFF 0x3E8 (1000) (1 second) 0x3E8 (1000) (1 second) No, needs to be added<br />
<br />
For more information about keep alives, see: http://support.microsoft.com/kb/120642<br />
<br />
976<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
28.6.3 Disable the System Tray network status icons In Windows there is an available option to show the status of a given network connection through a system tray icon. Each time there is network activity or interruption, the icon indicates this is the case. Because the icon blinks every time there is network activity and because a terminal server transmit screen updates information in a terminal server session an infinite loop occurs, the icon remains perpetually on and a constant stream of data (however small) is being sent down the network link. On an expensive or congested WAN link, this situation might well present a problem. To disable the system tray icon, follow these steps: 1. Open the Network Connections Control Panel. 2. Right-click the network interface in question, and choose Properties from the drop-down dialog. 3. Deselect Show icon in notification area when connected. 4. Click OK and exit.<br />
<br />
28.7 Load balancing Load balancing provides a level of high-availability that is often implemented in a much simpler manner than a full server clustered solution. For environments where data is static, it can provide an excellent and easy solution to offering greatly improved availability and performance than a single stand-alone server can deliver.<br />
<br />
28.7.1 Network Load Balancing You can use the Network Load Balancing (NLB) in Windows Server 2003 to provide high-availability and scalability to produce a single group of up to 32-nodes. To help with a thin-client environment, you can use NLB to distribute workload between multiple Terminal Servers. NLB can be particularly useful for environments where two or more Terminal Servers host the same, static server configuration and data. Network Load Balancing works by representing a group of servers by one virtual IP address so many servers with similar functions can be grouped together to support larger numbers of users. A new feature of Windows Server 2003 Network Load Balancing is the Session Director, a Windows service that keeps a list of sessions indexed by user name.<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
977<br />
<br />
This allows the user to reconnect to the server where their disconnected session resides and continue working with their applications. Session Directory prevents the user from connecting to another server in the farm and having independent, unassociated sessions running on multiple terminal servers. It is ideally efficient for server performance and offers a better experience for the user.<br />
<br />
File/print servers<br />
<br />
SQL Server/ Exchange servers<br />
<br />
Windows 2000 Terminal servers with NLBS<br />
<br />
Windows-based terminals or PCs<br />
<br />
Figure 28-8 Windows Server 2003 Terminal Services with load balancing<br />
<br />
28.7.2 Citrix Presentation Server Overall performance and availability is improved in multi-server environments by using the Citrix Load Balancing Services. Using the Windows Notepad application (notepad.exe) as a trivial example, if you decide to publish Notepad to your users, you can make it available on two or more servers. Then, when a user wants to use the Notepad application, the user’s client asks a server, called the Data Collector, to identify those servers that are running the application. The Data Collector (DC) then determines which server is the least busy of those hosting the application. It notifies the user’s client software with the name of the server to which it should connect to run the application. You can configure specific load evaluator rules for each application based on parameters such as server user load, application user load, CPU, and memory utilization. This all occurs without the user being aware of it. Citrix have recently released a global load-balancing product to provide a highly available solution for providing published application access across (widely) distributed multi-site Presentation Server deployments.<br />
<br />
978<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
28.7.3 Load-balancing versus high-availability Note that load-balancing when supplied either by Windows Network Load Balancing or Citrix load-balancing services should not be considered a true option for high-availability or HA. High-availability schemes generally drive for automatic failover to a secondary system if the production system is to fail. This generally occurs completely transparent to any users on the system at the point of failure. Clustering technologies typically with a storage-area-network (SAN) shared quorum disk are at the heart of such HA solutions. Load-balancing in its various iterations for Terminal Servers does not supply true high availability because if a given terminal server fails, the user’s session is interrupted and disconnected. Load-balancing provides improved performance for the user through balancing user sessions across multiple terminal servers. Load-balancing also offers an inexpensive, higher level of redundancy than a simple stand-alone server can offer, without the full complexity and overhead of server clustering. In the instance of a server failure, the user simply needs to start another connection to one of the servers in the load-balanced farm. This should happen completely automatically. If the system has been architected properly, all application and user data should be stored on another server, meaning that when the new terminal server session with the actual presentation layer of the application itself is established, the user can continue largely where they left off.<br />
<br />
28.8 Monitoring Windows Terminal Services Measuring the performance of your terminal server is similar to recording the performance of the underlying operating system. However, the counters of interest and their baseline values are different. You might choose to use the counter logs in the Performance console to record the server performance during a system pilot. Record your log files as comma-delimited text files to easily import them into such tools for further analysis as Microsoft Excel. Some counters, such as Terminal Services Session, record data only while a user is logged on to the terminal server. If a user logs off and their session ends, data is no longer recorded. Therefore, it is important to monitor who is logged on during the pilot and to start new Counter Logs if and when required. Table 28-2 shows the typical counters of interest for Terminal Services.<br />
<br />
Chapter 28. Microsoft Windows Terminal Services and Citrix Presentation Server<br />
<br />
979<br />
<br />
Table 28-2 Windows Server 2003 and Terminal Server counters<br />
<br />
Object and counter<br />
<br />
Instance<br />
<br />
Description<br />
<br />
What to see for acceptable user performance in the pilot...<br />
<br />
---<br />
<br />
The current number of active users accessing the terminal server. This should be plotted against the other performance values so you can extrapolate your servers performance.<br />
<br />
You need a minimum of 10 active users so that you can extrapolate some meaningful performance data.<br />
<br />
Terminal services Active Sessions<br />
<br />
Terminal services session % Processor Time<br />
<br />
Remote Desktop Protocol (RDP) or ICA session number1<br />
<br />
The amount of Processor Time for each user session. This per user value can assist you in determining whether one user’s set of applications is more CPU intensive than another.<br />
<br />
A small value as a percentage of the total % Processor Time. A larger value for a particular user indicates one user is running a particular CPU intensive application.<br />
<br />
Total Bytes (per second)<br />
<br />
RDP or ICA session number1<br />
<br />
The ICA or RDP network bandwidth per user transmitted and received on the network between the client and server.<br />
<br />
The amount of bandwidth varies depending on the graphical nature of the users session.<br />
<br />
You can use Excel to multiply each value by 0.008 to obtain a Kbps value so you can determine terminal server wide area network (WAN) requirements. Working Set<br />
<br />
RDP or ICA session number1<br />
<br />
The approximate amount of bytes used by each user. Use Excel to convert each value into a MB value so you can interpret more easily.<br />
<br />
The more programs a user runs, the larger the per session Working Set is. You can use this per user value to determine your typical users’ memory requirements.<br />
<br />
Note 1: Each time a user logs on to Windows Terminal Services, they are allocated a new session number.<br />
<br />
980<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
29<br />
<br />
Chapter 29.<br />
<br />
Microsoft Internet Information Services Windows Server 20031 includes Internet Information Server (IIS) 6.0, which supports the .NET Framework Applications and Web services. It also offers improved performance on existing ISAPI and ASP/COM+ applications. While IIS is usually regarded as a Web server, it also provides several other services, such as SMTP, FTP, and NNTP. Applications such as Exchange Server take advantage of these services and use these services instead of implementing them separately. This chapter focuses on the IIS HTTP server and includes the following topics: <br />
<br />
1<br />
<br />
29.1, “Introduction” on page 982 29.2, “Tuning IIS 6.0” on page 982 29.3, “Hardware settings” on page 1002 29.4, “Application tuning” on page 1009 29.5, “Monitoring performance” on page 1013 29.6, “Network load balancing” on page 1016<br />
<br />
Product screen captures and content reprinted with permission from Microsoft Corporation.<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
981<br />
<br />
29.1 Introduction The biggest challenge for tuning IIS is the large number of possible scenarios. IIS deployments range from machines that serve only a small set of static files to e-commerce solutions that require database-driven dynamic pages and distributed COM+ applications. Because of this diversity, providing guidelines for IIS is challenging. When you analyze bottlenecks on IIS, you must always consider the type of workload that the machine handles. The Web Capacity Analysis Tool (WCAT) V5.2 is used to simulate workloads on client/server configurations. You can use WCAT to test how your server that is running IIS 6.0 and your network configuration respond to a variety of client requests for content, data, or Web pages. You can use the test results to determine the optimal server and network configuration for your environment. The WCAT is available in the IIS 6.0 Resource Kit. It can run on Windows 2000 and later. You can download the IIS 6.0 Resource Kit Tools from the Microsoft Web site at: http://www.microsoft.com/downloads/details.aspx?FamilyID=56fc92ee-a71a4c73-b628-ade629c89499&DisplayLang=en<br />
<br />
29.2 Tuning IIS 6.0 With IIS 6.0, the process model is as follows. A kernel mode HTTP listener (http.sys) receives and routes HTTP requests. Worker processes register for URL subspaces, and http.sys routes the request to the appropriate process or a set of processes, in the case of application pools. The http.sys process is responsible for connection management and request handling. The request can either be served from the http.sys cache or handed to a worker process for further handling (see Figure 29-1 on page 983). You can configure multiple worker processes, providing isolation at lower cost. The http.sys process includes a response cache. When a request matches an entry in the response cache, http.sys sends the cache response directly from kernel-mode. Figure 29-1 shows the request flow from the network through http.sys. The kernel-mode response cache (enabled by default) can include a maximum of 64 GB of information about an x86 system and 512 GB on Windows Server 2003 64-bit edition. As single point of contact for all incoming (server-side) HTTP requests, http.sys provides connection management, bandwidth throttling, and Web server logging functions. IIS 6.0 is tuned for high Web server throughput. Static file requests, ISAPI components, and Microsoft ASP.NET pages can all<br />
<br />
982<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
cache response in the kernel-mode response cache. A caveat is that http.sys can only cache responses to HTTP GET requests in the kernel-model cache. Because no application code runs in http.sys, it cannot be affected by failures in user mode code that normally affect the status of the Web service. If an application fails, http.sys continues to accept and queue new requests on the appropriate queue until one of the following: <br />
<br />
The process has been restarted and begins to accept requests There are no queues available There is no space left on the queues The Web service itself has been shut down by the administrator<br />
<br />
After the WWW service notices the failed application, it starts a new worker process if there are outstanding requests still waiting to be serviced for the worker processes application pool. Thus, while there might be a temporary disruption in user mode request processing, a user does not experience the failure, because requests continue to be accepted and queued.<br />
<br />
Worker process Worker process Worker process<br />
<br />
Http.sys Request<br />
<br />
Request<br />
<br />
Send response<br />
<br />
Namespace HTTP engine<br />
<br />
Request<br />
<br />
Response cache<br />
<br />
REQUEST<br />
<br />
RESPONSE<br />
<br />
Figure 29-1 Request handling in IIS 6.0<br />
<br />
29.2.1 Worker process isolation mode Figure 29-2 illustrates worker process isolation mode, where each application runs in an isolated environment (w3wp.exe). On IIS 5.0, the user mode process first pulls the requests from kernel to application and then routes them accordingly to another user mode process. On IIS 6.0, http.sys routes Web site<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
983<br />
<br />
and application requests to the correct application pool queue. Then, the worker processes that serve the application pool pull requests directly from the application queue in http.sys.<br />
<br />
WWW service administration and monitoring component<br />
<br />
In worker process isolation mode, administrators can isolate different Web applications or Web sites into separate application pools. Each application pool is separated from other application pools by process boundaries. An application that is assigned to one application pool is not affected by other application pools, and cannot be assigned to another application pool while being serviced by the current application pool.<br />
<br />
Applications<br />
<br />
Applications<br />
<br />
Applications<br />
<br />
Worker process (W3wp.exe)<br />
<br />
Worker process (W3wp.exe)<br />
<br />
Worker process (W3wp.exe) User Kernel<br />
<br />
HTTP.sys<br />
<br />
Figure 29-2 IIS 6.0 worker process isolation mode<br />
<br />
You must tune two modes for an optimum performance: kernel mode and user mode. Note: Changing settings or registry keys can improve performance but can instead create bottlenecks and decrease performance. We recommend that you make the changes in a test environment and measure how the changes affect the system before implementing those changes in a production environment.<br />
<br />
29.2.2 Kernel mode The kernel mode tuning consists of changing http.sys parameters. These can be split into two categories: Cache management settings Request and connection management settings<br />
<br />
984<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Cache management settings With the kernel-mode cache, it is possible to satisfy an HTTP request entirely from kernel mode if the response is in the kernel cache, reducing the CPU cost of handling the request. On the other hand, the kernel-mode cache of IIS 6.0 is a RAM-based cache, and the cost of an entry is the memory it occupies. You can increase the Web server performance significantly by tuning the http.sys cache. The following registry keys are relevant: UriEnableCache HKLM\System\CCS\Services\Http\Parameters\UriEnableCache The default value for this key is 1. Any value different from 0 enables the kernel-mode response and fragment cache. This is the correct value for the most workloads. You should disable it only if you expect a small number of responses and fragmented utilization. UriMaxCacheMegabyteCount HKLM\System\CCS\Services\Http\Parameters\UriMaxCacheMegabyte The default value for this key is 0. Any value different from 0 sets the maximum memory available to the kernel cache. The recommended value is 0, which allow the system to automatically adjust the amount of memory available to cache. UriMaxUriBytes HKLM\System\CCS\Services\Http\Parameters\UriMaxUriBytes The default value for this key is 262144 bytes (256 KB). This is the maximum size of an entry in the kernel cache. The recommended value depends what kind of system do you have. The default value is good for most of the cases, but, If you have enough memory, increase this value and if memory is limited, and large entries are crowding out smaller ones, decrease this value. UriScavengerPeriod HKLM\System\CCS\Services\Http\Parameters\UriEscavengerPeriod The default value for this key is 120 seconds. A scavenger scans the http.sys cache each period of time set by this value. Entries not accessed during the scavenger period are removed. You have to adjust this value according with system behavior. If you set to a higher value, you reduce the scavenger scans, however the cache memory usage can grow as older, because you might have unused entries in the cache. If you set to a lower value will cause more scavenger scans which will might result in excessive flushes and cache churn. Start with default value in a test environment and change the value monitoring the system to reach the best value for your system.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
985<br />
<br />
UriMaxCacheUriCount HKLM\SYSTEM\CCS\Services\Http\Parameters\UriMaxCacheUriCount This registry entry determines how many responses can be cached. If zero, there is no limit. Range is 0x0 to 0xFFFFFFFF.<br />
<br />
Request and connection management settings Http.sys manages inbound HTTP/HTTPS connections and is the first layer to handle requests on these connections. It uses a reserve that is an internal data structure used to keep information about requests and connections. These reserves helps http.sys handle fluctuations in load with less CPU usage. The reserves help reduce CPU usage and latency, and increase Web server capacity but also increase memory usage. When tuning the request and connection management behavior of http.sys, it is important to keep in mind the resources available to the server, performance goals, and the characteristics of the workload. Use the registry settings bellow to manage MaxConnections resources. HKLM\System\CCS\Services\Http\Parameters\MaxConnections This key controls the number of concurrent connections http.sys will allow. Connection consumes non-paged-pool, a limited resource. The default is determined quite conservatively to limit the amount of non-paged-pool used for connections. On a dedicated Web server with ample memory, the value should be set higher if a significant concurrent connection load is expected. A high value might result in increased non-paged-pool usage, so use a test environment to monitor the system with different values to reach the best value and avoid a lack of resources.<br />
<br />
29.2.3 User mode The user mode handle with applications and work process are divided into three categories: “IIS registry settings” on page 987 “IIS metabase settings” on page 989 “IIS worker process options” on page 997<br />
<br />
986<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
IIS registry settings You can increase the Web server performance by modifying or adding these registry keys: MaxCacheFileSize HKLM\System\CCS\Services\Inetinfo\Parameters\MaxCachedFileSize This key is not included to the registry by default. The default value for this key is 256 KB (the range is unlimited). This key lets you specify the maximum size of the files to be cached. If your server has files in its content directories larger than this setting, these files will not be cached. If you have large Web servers with a lot of large frequently requested files, you should add this value to the registry to increase the size to a number that better fits your server necessity. Caching these files reduces the CPU usage, the disk access, and associated latencies. The best value depends on the size of the files and the number of requests. You have to monitor your Web server to determine the best value for this key. MemCacheSize HKLM\System\CCS\Services\Inetinfo\Parameters\MemCacheSize This key specifies the maximum amount of memory that IIS uses for its file cache. It is not included in the registry by default. The default value is approximately one half the available physical memory, which is calculated dynamically every 60 seconds. The range available varies from 0 to 2500 MB. The default value is good for most of the Web servers, but, for large Web Servers, consider adding this value to the registry to increase the amount of memory that IIS can use. DisableMemoryCache HKLM\System\CCS\Services\Inetinfo\Parameters\DisableMemoryCache The default value is 0. If you set this value to 1, you will disable the user mode IIS cache, static file caching is disabled. In all your production servers this parameter have to be set to 0. Disable this parameter could be useful during debugging, but on a production environment, if can considerably affect the servers performance. MaxPoolThreads HKLM\System\CCS\Services\Inetinfo\Parameters\MaxPoolThreads The default value for this key is 4. It sets the maximum number of pool threads that can be created per processor (range unlimited). Each pool thread watches for network requests and then processes them. It refers only to the number of worker threads available to process request for static files, not including threads that are currently processing ISAPI applications.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
987<br />
<br />
The default value is good for most of the cases, however, this parameter should be increased if the CPU shows sub-optimal average usage because all existing threads are busy and there is no available thread to process new requests. Note that the total number of IIS worker threads can be no greater than PoolThreadLimit. PoolThreadLimit HKLM\System\CCS\Services\Inetinfo\Parameters\PoolThreadLimit This key sets the maximum number of pool threads (range unlimited) that can be created in the inetinfo.exe, which limits the number of simultaneous connections. This parameter is not included in the registry by default. In its absence, the system calculates the value as two times the number of megabytes of RAM installed in the machine or a maximum of a maximum of 512. Adding this key overrides the calculated value. PoolThreadLimit must be greater than or equal to MaxPoolThreads. Normally PoolThreadLimit = MaxPoolThreads x number of processors. Setting only one of these parameters is sufficient. If both MaxPoolThreads and PoolThreadLimit are specified, the more stringent limit is used. ObjectCacheTTL HKLM\System\CCS\Services\Inetinfo\Parameters\ObjectCacheTTL This controls the length of time that objects are allowed to stay in the IIS user mode cache without being accessed. By default, this value is not added to registry. In its absence, the value defaults to 30 seconds. If you set the value to 0xFFFFFFFF, you will disable the scavenger thread. Disabling the cache scavenger is useful if your server has ample system memory and your data is relatively static. You can increase this parameter if there is no memory pressure on the system and if content that is served is static. Lower the parameter if the system is under memory pressure and the user mode cache is growing. ActivityPeriod HKLM\SYSTEM\CCS\Services\InetInfo\Parameters\ActivityPeriod The IIS 6.0 file cache uses an activity-period cache algorithm that attempts to cache only frequently requested files. The algorithm requires that a file be requested at least twice in an activity period (10 seconds by default), or the file will not be cached. If set the value of ActivityPeriod to 0, IIS always caches files. If an item in the user mode static-file cache is not access with 30 seconds (ObjectCacheTTL default), the item will be dropped from the user mode static-file cache. Range is 0-0xFFFFFFFF.<br />
<br />
988<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
DataSetCacheSize HKLM\SYSTEM\CCS\Services\InetInfo\Parameters\DataSetCacheSize Each IIS worker process (w3wp.exe) includes a local metabase data cache of up to 50 entries (by default). For each request, the required dataset is obtained from the local metabase cache if possible. If the dataset is not found in the cache, then an expensive LPC request is made to the Inetinfo.exe process for the dataset residing in the metabase. When there are more than 50 virtual directories, IIS disables the metabase data cache, forcing all requests to go through the expensive path (which negatively impacts throughput). This registry entry set the maximum number of dataset entries in the cache if the number of total installed virtual directories exceeds the default (50). For performance reason, we recommend that you turn off process recycling and pinging in IIS 6.0 to reduce unessential processes.<br />
<br />
IIS metabase settings The metabase is a repository for most IIS configuration values. The metabase is an XML file, Windows\System32\InetSrv\metabase.xml. You can edit it manually or use Metabase Explorer, which is part of the IIS 6.0 Resource Kit that is available from: http://www.microsoft.com/downloads/details.aspx?FamilyID=56fc92ee-a71a4c73-b628-ade629c89499 Tip: Metabase Explorer replaces MetaEdit, which is available from: http://support.microsoft.com/?kbid=232068 The following metabase values have an impact on IIS performance: AppAllowDebugging Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
989<br />
<br />
This property specifies if ASP debugging is enabled on the server. The default value is false (disabled). When you enable debugging, only one thread is allowed to execute at a time for each application affecting the Web server performance. We strongly recommend that you do not change the default on production servers. AspBufferingOn Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This property specifies that all output from an application be collected in the ASP output buffer before the buffer is flushed to the client browser. The default value is true, which we recommend for all production servers. If you set this property to false, the output from ASP scripts are written to the client browser as the browser becomes available, which will appear to be faster to the user. This is true for the first client, but for the next clients the response will be slower. AspDiskTemplateCacheDirectory Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This property sets the name of the directory that ASP uses to store compiled ASP templates to disk after overflow of the in-memory cache. The default value is %windir%\system32\inetsrv\asp. For performance reasons, ensure that this location is on a drive that does not compete with the IIS log, operating system pagefile, or frequently accessed content.<br />
<br />
990<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
AspExecuteinMTA Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This is a useful setting if components are primarily free-threaded or both free-threaded and apartment-threaded. Setting this property to 1 (default is 0) allows the running of all its threads in a multi-threaded apartment (MTA). Free-threaded and apartment-threaded: A thread is a parallel execution of part of the program in the same memory space. The free-threaded COM objects are characterized by the fact that they are designed to process calls from different threads without help from the COM library. The other possible way is apartment-threaded, where the calls are handle by the COM library. AspMaxDiskTemplateCacheFiles Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This property is used to allow disk caching of ASP script templates. We recommend that you leave this property in this default value 2000. Compiling ASP templates is a processor-intensive task and memory restricts the number of templates that can be cached in memory.Range is 0-0xFFFFFFFF.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
991<br />
<br />
AspProcessorThreadMax Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This property specifies the maximum number of worker threads per processor that IIS can create. It defines the maximum ASP requests that can execute at the same time. We strongly recommend that you leave this property in the default value (10). If your application makes extended calls to external components, you should consider increase this value, but, you should do that on a test environment and monitor the performance before set the new value to this property. AspQueueConnectionTestTime Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This is a very important property for the Web server performance. It allows the server to check whether the user are still connected before beginning a task execution. It is important because sometimes an impatient user submits the same request several times and move to another site before the server‘s response leaving a lot of lost requests in the queue for that application pool. The default value is three seconds, which is the recommended value.<br />
<br />
992<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
AspRequestQueueMax Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This property specifies the number of ASP requests that can be allowed in the queue. The default value is 500. The ASP requests are responded to, they are removed from the queue. When this maximum is reached, clients get the following message: HTTP 500 Server Too Busy You have to monitor the server performance to set the best value for the queue. If requests stay in the queue for a short time, you could consider increasing this value. We recommend 2000 for most of the servers. AspScriptEngineCacheMax Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This property sets the maximum number of scripting engines that ASP pages keeps cached in memory. The default value is 120 pages. The best practice is to adjust the value according to the type of content in the application. If you have thousands of unique pages you should increase this cache size, so the most frequently requested pages can be accessed quickly.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
993<br />
<br />
AspScriptFileCacheSize Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This property specifies the number of precompiled script files to cache. The default value is 500. If you set the value to 0xFFFFFFFF, all script files are cached. You have to measure the amount of available memory and the amount of script file traffic to set the best value. The ASP template cache uses a least-recently-used algorithm for determining which templates are cached. This means that if the cache is full, the template that has been in the cache and has not been requested for the longest amount of time is replaced by the next template to enter the cache. Never set this value to zero because this turns the cache off and has an impact on the server‘s performance. If you have servers with sites with a small number of requests, set this property to a small value, so that you can gain more system memory. AspSessionMax Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This property specifies the maximum number of concurrent sessions that IIS will permit. The default value is 4294967295 (0xFFFFFFFF). It is appropriated<br />
<br />
994<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
to set a lower value to limit the memory overhead. If a client attempts a new session after the limit is reached, it receives the following error: HTTP 500 Server Too Busy AspSessionTimeout Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This property specifies in minutes the amount of time that a Session object is maintained after the last request associated with the object is made. Because the session consumes memory, it is appropriate to limit the session lifetime. The default value for this property is 10. Consider decreasing this value if the memory overhead is too high. AspTrackThreadingModel Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory This property specifies whether IIS checks the threading model of any components that your application creates. The default value is false. We strongly recommended that you leave the default value. If you set this value to true, any component that your application instantiates is checked, which creates an impact on your server‘s performance.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
995<br />
<br />
DisableLazyContentPropagation HKLM\SYSTEM\CCS\Services\ASP\Parameters\DisableLazyContentPropagation Default value: 0 (lazy propagation enabled) Range: 0 - 1<br />
<br />
Lazy propagation refers to the action that IIS takes when a large amount of content is updated at one time. IIS has an internal limit on the amount of content that can be updated in the in-memory template cache. If the size of the updated content exceeds that limit, IIS marks each of the files in the in-memory template cache as invalid. When it receives the first request to an invalid file, IIS begins to compile a new template, but it serves the expired template until the new template is compiled. If the value of DisableLazyContentPropagation is set to 1 (the default is 0), IIS behaves as it does for IIS 5.0 and IIS 5.1 when a large amount of content is updated at one time. IIS flushes the in-memory template cache, and performance can slow to a standstill as each new request to the server forces IIS to compile new templates. We recommend that you use to the default value of 0. Valid values are 0 and 1 only. CacheISAPI Paths: – /LM/W3SVC/n/ROOT to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n/ROOT/virtual_directory_name to WebVirtualDir (IIS Admin Object Type) – /LM/W3SVC/n to IIsWebServer – /LM/W3SVC to IIsWebService – /LM/W3SVC/n/ROOT/physical_directory_name to IIsWebDirectory – /LM/W3SVC/n/virtual_directory_name/physical_directory_name to IIsWebDirectory The CacheISAPI property indicates whether ISAPI extensions are cached in memory after first use. The default value is true. If the value of this property is true, the DLL file remains in the cache until the server is stopped. If the value is false, ISAPI extensions are unloaded from memory after the extension DLL is no longer in use. The false value is recommended only for test environments, because reloading an ISAPI extension DLL file for each request impacts a server‘s performance significantly. CentralBinaryLoggingEnabled Path: /LM/W3SVC to IIsWebService This property centralizes the IIS log file in a single binary file instead of a separate log for each Web site. The default value is false. If you do not need<br />
<br />
996<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
a separated log for each Web site, consider setting this property to true to reduce CPU usage, disk usage, and disk I/O.<br />
<br />
IIS worker process options There are several settings that is available to tune the applications pool. These are accessed through the properties of the application pool, as follows: 1. In IIS Manager, expand the host (for example, the local computer), Application Pools, and the relevant application pool. 1. Right-click the application pool that you want to configure and click Properties. 2. Make the changes as discussed in the following sections and click OK to save the changes: – “Recycling options” on page 997 – “Performance options” on page 999 – “Health options” on page 1001<br />
<br />
Recycling options The IIS 6.0 has some recycling options to restart the worker process that is assigned to an application pools. These options can prevent a system crash because the applications with problems are restarted before any instability in the application or in the system.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
997<br />
<br />
To force a worker process to recycle immediately, right-click the application application pool that you want to restart immediately and click Recycle. You configure recycling options in the Recycling tab, Figure 29-3.<br />
<br />
Figure 29-3 Recycling tab of the Application Pool properties<br />
<br />
You can select one or more ways of specifying when to recycle worker processes: Select Recycle worker processes (in minutes) and enter the number of minutes that you want between recycles. Select Recycle Worker Processes (number of requests) and enter the number of requests that you want between recycles. Select Recycle Work Process at the following times and use Add to add set times. Select Maximum virtual memory (in megabytes) or Maximum used memory (in megabytes) and enter a suitable value to configure the worker process to recycle after consuming a set amount of memory (virtual or physical).<br />
<br />
998<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Performance options The Performance tab (Figure 29-4) lets you specify various options.<br />
<br />
Figure 29-4 Performance tab of the Application Pool properties<br />
<br />
From this tab, you can set the following: Shutdown the worker process after being idle for a set amount of time It is important to clean up the memory that is used for the worker process when a badly written application does not release the resources. Note that after you shutdown a worker process, it is necessary to generate a new one in the next request. So, we do not recommended that you activate this parameter if the application needs to generate a worker process frequently. If you activate this parameter, it can create a CPU bottleneck due to the overhead that is associated with process creation. The default value is 60 minutes, which is the recommended value. Request Queue Limit Enter the number of requests that you want to allow before the server stops to respond the requests. Recycling the worker process avoids system crashes. If you do not set a queue limit, when the requested queue reaches a high value, IIS ceases to respond, and you have to restart the IIS service.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
999<br />
<br />
A request consumes paged-pool size. Start with a value, for example 100, and increase this value if necessary. After the queue reach the value specified, the server rejects the connection and issues Error 503 to the client. Enable CPU Monitoring This property monitors the CPU usage and, depending on the values that you set, writes an event log or even shuts down the worker process You can specify the maximum CPU utilization before an event is triggered: – Maximum CPU use (percentage): Set the maximum CPU usage allowed before the action is taken – Refresh CPU usage numbers (minutes): enter how often to check the CPU utilization – Action performed when CPU usage exceeds maximum CPU use: specifies what action you want to take. Choices are: • •<br />
<br />
No action: Writes and event to the Event Log Shutdown: shuts down the worker process<br />
<br />
Web garden Web garden allows several worker processes handle the request load under a single application pool. It alleviates the effect of an application that has software contention problems by indirectly creating multiple instances of the resource and HTTP.sys distributes the load among the worker processes in a round-robin order. The default value is one worker process for each application pool. You should increase the maximum number of worker processes if you see you have poor CPU usage, but the response time is long, because that means that the number of worker processes is not large enough to handle the request load. So, monitor the CPU usage and the response time to choose the best number for this property.<br />
<br />
1000<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Health options The Health tab of the Properties dialog (Figure 29-5) has some proactive settings that you can use to monitor the worker processes health and take an action.<br />
<br />
Figure 29-5 Health tab of the Application Pool properties<br />
<br />
Options in this tab are: Enable pinging It is very important to enable the worker process pinging. Enabling this property will allow IIS to terminate the failure worker process and start a new one. Start with the default value and monitor the system. If you do not experience performance degradation, keep this value. Otherwise, increase the pinging time. Enable rapid-fail protection This is a setting to control an error control of the application pool. You set the maximum number of failures in a amount of time for the application full and allow the system to disable this application pool. Startup and Shutdown Time Limit This two properties are the maximum amount of time before an application pool is considered to have failed during startup and shutdown. The default value is 90. We recommend that you use the default value.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
1001<br />
<br />
29.3 Hardware settings Most of the tuning activity for IIS is performed at the application level; however, there are some recommendations and settings that can help to improve the Web server performance. The most relevant hardware subsystems for IIS performance are: Network CPU Memory<br />
<br />
29.3.1 Network subsystem The network is often the bottleneck for IIS server deployments, usually due to bandwidth limitations of the server’s connection to the Internet, rather than the network interface card of the server itself.<br />
<br />
HTTP compression Enabling HTTP compression can improve IIS performance because IIS compresses a file when it is requested for the first time and these compressed files then are sent on subsequent requests. The result is that less data is sent, meaning a lower network cost and an increase in network throughput. For files that provide a good compression ratio, such as text or HTML, the increase can be as much as 400% in network throughput. If the network bandwidth is limited, HTTP compression can be beneficial, unless CPU utilization is already very high. HTTP compression can improve on latency of responses as well as network throughput. You can compress either static or dynamics files but you need to take care about dynamic content because dynamic content is not cached to disk, so the content is effectively compressed in every request and this can impact CPU resources. We recommend compressing only static content. To enable the HTTP compression feature: 1. In IIS Manager, expand the local computer, application pool. 2. Right-click Web sites and click Properties. 3. Click the Service tab (Figure 29-6). 4. Deselect Compress application files for dynamic content. (This option is not recommended.) 5. Select Compress Static Files for static content. This option is recommended.<br />
<br />
1002<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
6. You can also specify the maximum amount of disk space that the HTTP Compression setting Maximum temporary directory size uses to unlimited or to a maximum value in megabytes.<br />
<br />
Figure 29-6 Enabling HTTP compression<br />
<br />
By default, only files with extension HTM, HTML, and TXT are compressed when compression on static files is enabled. To add other static file types such as CSS, JS, and XML, execute the following commands at a command prompt: cscript %SystemDrive%\Inetpub\AdminScripts\adsutil.vbs set W3SVC/Filters/Compression/deflate/HcFileExtensions "htm html txt css js xml" cscript %SystemDrive%\Inetpub\AdminScripts\adsutil.vbs set W3SVC/Filters/Compression/gzip/HcFileExtensions "htm html txt css js xml" You can use dynamic content compression if latency for users is an issue and the extra CPU utilization, because IIS6.0 must run its compression routine on every response, is acceptable.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
1003<br />
<br />
HTTP keep-alive HTTP keep-alive greatly reduces the cost of client-server communication, because connections between browser clients and the Web server are kept open after they are established. HTTP keep-alive settings are enabled by default, and generally should not be turned off. To enable HTTP keep-alive, do the following: 1. In IIS Manager, expand the local computer, application pool. 2. Right-click Web sites and click Properties. 3. On the Web Site tab (Figure 29-7), under Connections, click Enable HTTP Keep-alive.<br />
<br />
Figure 29-7 Enabling HTTP keep-alive<br />
<br />
29.3.2 CPU subsystem The load on the CPU subsystem is largely dependent on the type of application that the Web server is running. For a system serving mainly static content, the bottleneck probably is the network rather than the processor. Alternatively, database-driven Web sites that implement business logic using ASP, ISAPI or<br />
<br />
1004<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
COM+ components, .NET can place heavy load on the CPU because pages are generated dynamically for each request. For most IIS deployments, the CPU subsystem is not a bottleneck. You should thoroughly check for bottlenecks in other subsystems, especially the memory subsystem, before adding processors to your Web server. There is, however, one specific type of environment where the CPU subsystem can become a bottleneck. If you are using Secure Sockets Layer (SSL) to encrypt the traffic between your site and your clients, a considerable amount of CPU cycles is required to encrypt and decrypt data. In addition, network throughput might decrease due to the overhead generated by using SSL. Expect network throughput to be at least 10 times lower than without SSL. If you support a large number of SSL clients on your site, ensure that the SSL session timeout matches the time users actually stay on your site. If you set the timeout to too small a value, the SSL session will expire, and the server and client will have to re-establish a secure connection channel by creating new encryption keys. This process takes a considerable amount of computing power (500% to 600% the amount of a non-SSL connection), so you should avoid setting the timeout too low. Alternatively, setting the timeout too high increases overhead because the server has to maintain connection data for clients that are no longer connected. To modify the default setting of five minutes, add the following DWORD parameter to the registry: HKLM\SYSTEM\CCS\Control\SecurityProviders\SCHANNEL\ServerCacheTime This value sets the timeout for the cache in milliseconds. The default setting is 300 000 milliseconds (5 minutes). For very large sites, consider using hardware-based SSL accelerators to offload the SSL overhead from the Web servers.<br />
<br />
Memory subsystem The memory subsystem is probably the most important subsystem for IIS operations. You achieve easy performance gains by resolving issues that are related to memory subsystem bottlenecks. While documentation states 128 MB as the minimum amount of RAM needed for IIS, it is recommended that you use at least 256 MB and up to 1 GB for most Web sites. More memory might be beneficial for sites with a large amount of content. The first step is to ensure that Windows Server 2003 is configured to treat the IIS server as an application server. With IIS, it might not seem obvious whether to specify the machine type as an application or file server as it is with other software such as Exchange or SQL Server. However, IIS performs best if the<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
1005<br />
<br />
server type is configured as an application server, even when it is just serving static content. To specify the type of server on a Windows 2003 machine, follow these steps: 1. Click Start → Control Panel → Network Connections. 2. Right-click any network connection and click Properties. 3. Select File and Print Sharing for Microsoft Networks and click Properties. 4. Select Maximize data throughput for network applications (Figure 29-8) and click OK. Tip: If you have a large amount of RAM installed (for example, 2 GB or more), you might want to use the Maximize data throughput for file sharing option instead. See 11.6.1, “Servers with large amounts of free physical memory” on page 314 for details.<br />
<br />
Figure 29-8 Adjusting file system cache settings<br />
<br />
1006<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Other adjustments that you can make that affect memory subsystem include: IIS file cache purging—ObjectCacheTTL See “ObjectCacheTTL” on page 988. IIS file cache size—MemCacheSize See “MemCacheSize” on page 987. Maximum size of files to cache—MaxCachedFileSize See “MaxCacheFileSize” on page 987. Ensure cache is not disabled—DisableMemoryCache See “DisableMemoryCache” on page 987.<br />
<br />
Disk subsystem The disk subsystem is usually the least concern for IIS applications. With enough memory installed, IIS will cache most of the content in memory, greatly reducing disk access. Even with most of the sites content being served from the memory cache, IIS still writes log files to disk. A successful Web site can create megabytes’ or even gigabytes’ worth of log files easily every day. For these kinds of loads, it can be advisable to locate the log files on a dedicated RAID-1 array. Also, you should configure IIS to create a new log file every hour instead of the default setting of once a day for high volume sites, because smaller files are easier to work with. Ideally, logs should be separated onto separate devices with separate controllers if possible to increase parallelism in logging. The goal is to split out the I/O for busy sites so that log buffer writes for one busy site are not queuing behind log buffer writes for a different site, with both being written to the same device. If this is the case, the overall logging process has a much higher latency. To determine the busy sites on a server, there are a few parameters you can monitor. In System Monitor, monitor the counter Web Service:Total Method Requests—select the specific instances of the sites you want to monitor. The Total Method Requests item displays how many requests the site has had since IIS was started. From here, you can deduce quickly which sites are the busy sites and plan the log file subsystem accordingly. If you do not need log files, you should disable logging for your site. If you need log files for user tracking or reporting, you still might want to disable logging for directories that store images or other static files, where logging would not be of value to you.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
1007<br />
<br />
To disable logging for a specific directory or site: 1. In IIS Manager, expand Web sites. 2. Right-click the Web site or directory for which you want to disable logging, and click Properties. 3. On the Web Site tab (Figure 29-9), deselect Enable logging.<br />
<br />
Figure 29-9 Disabling logging for a specific directory or site<br />
<br />
29.3.3 Other performance tuning factors In addition to IIS tuning, TCP/IP and NIC tuning also have a major impact on Web server performance. Process affinity, where worker processes can be affinitized to CPUs to run on specified CPUs, can also take advantage of CPU cache (L1 and L2) hits.<br />
<br />
1008<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
29.4 Application tuning Besides optimizing the application code itself, you can improve the performance of ASP applications by modifying IIS cache settings and disabling features that you do not need to run your application. Because some of the following settings have an impact on ASP development, you should check with your developers or your software vendor before modifying these parameters. Application tuning that can improve performance includes the following: Disable IIS session state By default, IIS provides a session management mechanism using cookies. Developers often use the session object in ASP to store data related to a specific client. For high-performance Web sites, it is recommended that you disable this feature. Using the IIS session object prevents a Web site from scaling gracefully, because it does not span multiple servers. Developers need to implement a different kind of session management that can be used in a load-balanced environment. Disabling IIS session state also increases performance on single servers that do not use the session object. Again, notify your developers before you disable this feature. To disable IIS session state: a. In IIS Manager, expand Web sites. b. Right-click the Web site that you want to change and click Properties. c. On the Home Directory tab, under Application Settings, click Configuration. d. On the Options tab, deselect Enable session state.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
1009<br />
<br />
Figure 29-10 Disabling sessions states<br />
<br />
Enable ASP buffering ASP buffering stores the output of an ASP page while it is created and only transfers the complete page to the client when processing is finished. This method minimizes the overhead that is needed for managing ASP output. Note: Users might actually regard the page as slower, because they see no output until processing is finished. Developers should use the response.flush method to release parts of the page early for pages that take a long time to finish. To enable ASP buffering: a. In IIS Manager, expand Web sites. b. Right-click the Web site that you want to change and click Properties. c. On the Home Directory tab, under Application Settings, click Configuration. d. On the Options tab, select Enable buffering.<br />
<br />
1010<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 29-11 Enabling buffering<br />
<br />
Enable ASP caching IIS stores precompiled copies of ASP pages in its cache to speed up subsequent requests. With a page cached, it does not have to compile it when it is requested the next time. You can set the number of cache files to be stored in memory and disk. The default value is 500 for memory and 2000 for disk. If your site uses a large number of individual ASP pages, increasing the number of files to be cached yields performance gains. Never disable ASP file cache because doing so impairs severely your server’s performance. Figure 29-12 on page 1012 shows the cache options. To change ASP file cache options: a. In IIS Manager, expand Web sites. b. Right-click the Web site that you want to change and click Properties. c. On the Home Directory tab, under Application Settings, click Configuration. d. On the Cache Options tab, set the file cache options that you want.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
1011<br />
<br />
Figure 29-12 ASP file cache options<br />
<br />
Disable ASP debugging On development machines, ASP debugging is often turned on to help during code testing and debugging. This option serializes all ASP requests, which means that only one request is run at a time, resulting in a severe loss of performance. Therefore, you should make sure that this option is turned off on all your production machines. By default, IIS 6.0 disables ASP debugging. To disable ASP debugging: a. In IIS Manager, expand Web sites. b. Right-click the Web site that you want to change and click Properties. c. On the Home Directory tab, under Application Settings, click Configuration. d. On the Debugging tab, deselect Enable ASP server side scripting debugging and Enable ASP client side script debugging.<br />
<br />
1012<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Figure 29-13 Disabling ASP debugging<br />
<br />
29.5 Monitoring performance IIS and ASP offer a large amount of specific performance monitor counters, which allow you to monitor your Web server in great detail. Keep in mind that monitoring and collecting data for all of these parameters can have an impact on performance. Use a test environment to simulate the load on the system before deployment. Note: For analyzing long-term trends, use the Capacity Manager tool that is part of IBM Director. Most of values provided in Table 29-1 are relative because the optimum values can vary with the Web application, system and network architecture. So, you should use this table as a baseline and adjust the target values according to your system needs.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
1013<br />
<br />
Table 29-1 IIS performance counters<br />
<br />
Counter<br />
<br />
Target value<br />
<br />
Comment<br />
<br />
Pages/sec<br />
<br />
0 to 20<br />
<br />
Values above 80% can indicate a lack of memory.<br />
<br />
Available Bytes<br />
<br />
10% of the physical memory or more<br />
<br />
Add more memory if this value is below 5% of the physical memory.<br />
<br />
Committed Bytes<br />
<br />
No more than 75% of physical memory<br />
<br />
Pool Nonpaged bytes<br />
<br />
A constant value<br />
<br />
A slow rise can indicate a memory leak.<br />
<br />
%Processor Time<br />
<br />
Less then 75%<br />
<br />
If this counter is high while network and disk I/O counters are not at high levels, the CPU is the bottleneck. On SMP systems, monitor each processor individually to detect imbalances on process distribution.<br />
<br />
Interrupts/sec<br />
<br />
As low as possible<br />
<br />
This value depends upon the processor, network hardware, and drivers.<br />
<br />
System Processor Queue Length<br />
<br />
4 or less<br />
<br />
Memory<br />
<br />
Processor<br />
<br />
Logical Disk %Disk Time<br />
<br />
As low as possible<br />
<br />
Avg. Disk Queue Length<br />
<br />
Less than 4<br />
<br />
Avg. Disk Bytes/Transfer<br />
<br />
As high as possible<br />
<br />
Physical Disk %Disk Time<br />
<br />
As low as possible<br />
<br />
Avg. Disk Queue Length<br />
<br />
Less than 4<br />
<br />
Avg. Disk Bytes/Transfer<br />
<br />
As high as possible<br />
<br />
1014<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Counter<br />
<br />
Target value<br />
<br />
Comment<br />
<br />
System Context Switches/sec<br />
<br />
Compare this value with the value of Web Service/Total Method Request/sec. Context switches per request (Context Switches divided by Total Method Requests/sec) should be low.<br />
<br />
System Calls/sec<br />
<br />
As low as possible<br />
<br />
Web Service Bytes Total/sec<br />
<br />
As high as possible<br />
<br />
Total Method Request/sec<br />
<br />
As high as possible<br />
<br />
Current Connections<br />
<br />
As high as possible<br />
<br />
File Cache Hits%<br />
<br />
As high as possible for static content<br />
<br />
If this value drops well below 75% on a server with a mainly static file workload, this indicates a memory bottleneck.<br />
<br />
Kernel:URI Cache Flushes<br />
<br />
As low as possible relative to the number of requests<br />
<br />
Every time a file is flushed from the http.sys response cache this number increase. frequent flushes cause http.sys to use more memory for content that is not being accessed. Flush the cache less often is a good practice.<br />
<br />
Kernel:URI Cache Misses<br />
<br />
As low as possible<br />
<br />
Each request from dynamic content increases the value of the counter by 1.<br />
<br />
Kernel:URI Cache Hits%<br />
<br />
As high as possible<br />
<br />
Applies to static unauthenticated content and dynamic content that is marked as cacheable.<br />
<br />
Requests/sec<br />
<br />
No ideal value. See comments<br />
<br />
This value gives an overview of the amount of work your server is performing. Watch this value over time to determine long-term growth.<br />
<br />
Requests Executing<br />
<br />
No ideal value<br />
<br />
Shows the number of concurrent requests. If this value is never higher than 1, check if debugging is turned on.<br />
<br />
Active Server Pages<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
1015<br />
<br />
Counter<br />
<br />
Target value<br />
<br />
Comment<br />
<br />
Request Wait Time<br />
<br />
As low as possible<br />
<br />
Displays the time in milliseconds that the last request had to wait in the queue before it was executed. This value should not be higher than a few seconds.<br />
<br />
Request Execution Time<br />
<br />
As low as possible<br />
<br />
Depending on the complexity of your ASP applications, this value can be up to several seconds. If this value is more than few seconds, consider redesigning your application logic into smaller parts to minimize user wait time.<br />
<br />
Requested Queued<br />
<br />
As low as possible (0 is the ideal)<br />
<br />
Analyze this value in conjunction with the Request Wait Time counters over time, to determine if there is a bottleneck. Even when this value is high (several hundred), a low Request Wait Time is an indication that your server is performing well.<br />
<br />
Transaction/sec<br />
<br />
As high as possible<br />
<br />
ASP transactions degrade overall server performance because each transaction requires interaction with a database. If you are concerned about server performance, use ASP transactions sparingly.<br />
<br />
29.6 Network load balancing The tuning options that we describe in this chapter are all for a single Web server. However, in many environments, a single server is not able to handle all the workload, and administrators have to deploy more servers for the same content (that is, a Web server farm). A simple and common solution for such environments is DNS roundrobin. If you have more IP addresses available, you can configure your DNS server so that it always responds with another configured IP address. For example, if you have 10 Web servers hosting the same content implemented in your infrastructure, the DNS server directs the first client request to go the first Web server, the second response is directed to the second Web server, and so on.<br />
<br />
1016<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
The advantage of roundrobin is that it distributes the number of client requests across all Web servers, but the disadvantages are: It only distribute the requests, not the workload. If a Web server is offline, the DNS distributes its IP address but the affected clients cannot connect to the other Web servers until they flush their DNS cache and request the IP for the Web site again from the DNS. If a Web server is taken offline or a new Web server is deployed, you must modify the DNS settings to include the new IP address. Windows 2003 supports the following cluster technologies: Server clustering (MSCS) Provides automatic failover of the workload of a failed server to another server in the cluster. It offers high availability but does not increase performance. Component load balancing (CLB) Performs dynamic load balancing over multiple nodes of middle tier application components that use COM+. Network load balancing (NLB) NLB offers scalability and high availability for Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Generic Routing Encapsulation (GRE) traffic requests. NLB from Microsoft is the recommended version to ensure high availability and scalability for Web server farms. NLB supports Web servers and other services such as virtual private network (VPN) servers and Internet Security and Acceleration (ISA) servers. Generally, NLB handles stateless applications where each client request is an independent operation and the load balance for each request can be handled independently.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
1017<br />
<br />
Figure 29-14 illustrates the NLB functionality using only one virtual IP address.<br />
<br />
Internet/intranet<br />
<br />
NLB virtual IP address<br />
<br />
... NLB Host<br />
<br />
NLB Host<br />
<br />
NLB Host<br />
<br />
NLB Host<br />
<br />
NLB Cluster<br />
<br />
Figure 29-14 NLB functionality<br />
<br />
The NLB service is configured on each of the servers in the group to share this virtual IP address. The company associates the fully qualified domain name (FQDN) of the Web site with the virtual IP address. If a client requests the DNS server to provide the IP of the Web site, the DNS responds with the virtual IP address of the cluster, and the client starts a request against the cluster IP. The servers in the NLB cluster negotiate with each other and if no other configuration is set, the server with the lowest load responds to the request. To accomplish this, NLB works with multicast MAC addressing. When a NLB cluster is created, a multicast MAC address is also created for the virtual IP address. So, if there is a request to the virtual IP address, the request is broadcast to the entire subnet of that IP. The servers in the cluster negotiate to see which has the lowest workload. The server with the lowest workload responds to the request. This functionality brings the following advantages: Servers can join or be taken offline from the cluster dynamically Up to 32 servers can join a NLB cluster The group members negotiate with the others and are able to distribute the workload in a way that increases performance Traffic is never sent to an offline server<br />
<br />
1018<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
High service availability is maintained NLB offers scalability to improve performance However, this method also has its disadvantages: It works at the network level, so if a server crashes, NLB removes the server automatically from the cluster. However, if the server remains online and only the application crashes, NLB continues to send traffic to the server. NLB works with multicast MAC addresses, so each packet that is addressed to the virtual IP is broadcast in the subnet, and only one of the servers handles the request. All others drop the request, which can cause high network usage. NLB requires CPU processing overhead. There is no setup required to install the NLB cluster service. NLB is implemented in the TCP stack. You have to enable it, as shown in Figure 29-15, and configure it with the Network Load Balancing Manager (NlbMgr).<br />
<br />
Figure 29-15 NLB driver<br />
<br />
To configure the driver, open a command prompt and type NlbMgr to start the manager.<br />
<br />
Chapter 29. Microsoft Internet Information Services<br />
<br />
1019<br />
<br />
A checklist that describes how to enable and to configure the NLB driver can be found at: http://go.microsoft.com/fwlink/?LinkId=18371<br />
<br />
29.6.1 NLB with Layer 7 switches The CPU processing overhead with IIS is incidental, and because of the increased performance and high availability that is reached by distributing the workload to several servers, most administrators are not concerned with that overhead. However, in some infrastructures, the multicast MAC addresses are an issue. If the increased network traffic in the subnet becomes too much, the response time suffers and the NLB cluster will not perform well. The best solution for high performance and high availability Web farms are Layer 7 Ethernet switches, such as the IBM Nortel Networks Layer 2-7 Switch Module for IBM BladeCenter. Layer 7 switches offer high throughput and alleviate the server CPUs from processing load balancing tasks because they forward the requests directly to the target server, which eliminates the broadcast issue. Layer 7 switches perform switching based on application data (for example http headers). Because of the information that is available on this level, they can detect application and server down conditions to enable fault tolerance and to provide sophisticated load balancing. For example, Web sites can have so much content that is associated with their domain name that the content needs to be split across multiple file systems. In this situation, you could allow each Web server access to each file system by cross-mounting all the file systems. However, this method gets unwieldy because the number of file systems gets larger if it routinely changes. Another approach is to assign access to portions of the directory space to certain Web server clusters but still advertise the site under one domain name, such as www.ibm.com. The switch, as a front end to this site, must be able to inspect the URL request (including file name and path name) and to send the requests for www.ibm.com/marketing/ to one server, www.ibm.com/research/ to another server, www.ibm.com/admin/ to another server, and so forth. This method is just one example of the applications that a Layer 7 switch can provide. You can find more information about Layer 7 switches in Application Switching with Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module for IBM BladeCenter, REDP-3589, which is available from: http://www.redbooks.ibm.com/abstracts/redp3589.html<br />
<br />
1020<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Related publications We consider the publications that we in this section particularly suitable for a more detailed discussion of the topics that we cover in this IBM Redbook.<br />
<br />
IBM Redbooks For information about ordering these publications, see “How to get IBM Redbooks” on page 1030. IBM System Storage Solutions Handbook, SG24-5250 Netfinity and Domino R5.0 Integration Guide, SG24-5313 DB2 UDB V7.1 Performance Tuning Guide, SG24-6012 Implementing IBM Director 5.10, SG24-6188 Using iSCSI Solutions' Planning and Implementation, SG24-6291 Implementing VMware ESX Server 2.1 with IBM TotalStorage FAStT, SG24-6434 Planning and Installing the IBM Eserver X3 Architecture Servers, SG24-6797 IBM TotalStorage Disk Solutions for xSeries, SG24-6874 DB2 UDB V8.2 on the Windows Environment, SG24-7102 Virtualization on the IBM System x3950 Server, SG24-7190 Running the Linux 2.4 Kernel on IBM Eserver xSeries Servers, REDP-0121 Application Switching with Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module for IBM BladeCenter, REDP-3589 Implementing Windows Terminal Server and Citrix MetaFrame on IBM Eserver xSeries Servers, REDP-3629 Introducing IBM TotalStorage FAStT EXP100 with SATA Disks, REDP-3794 IBM Eserver BladeCenter and Topspin InfiniBand Switch Technology, REDP-3949 VMware ESX Server: Scale Up or Scale Out?, REDP-3953 Introducing Windows Server x64 on IBM Eserver xSeries Servers, REDP-3982<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
1021<br />
<br />
SQL Server 2005 on the IBM Eserver xSeries 460 Enterprise Server, REDP-4093 Domino 7 Performance Tuning Best Practices to Get the Most Out of Your Domino Infrastructure, REDP-4182 ServeRAID Adapter Quick Reference, TIPS0054<br />
<br />
Referenced Web sites These Web sites are also relevant as further information sources:<br />
<br />
IBM Web sites IBM support site http://www.ibm.com/servers/eserver/support/xseries/ IBM TotalStorage http://www.ibm.com/storage xREF spec sheets http://www.ibm.com/servers/eserver/education/cust/xseries/xref.html IBM Chipkill Memory white paper http://www.ibm.com/systems/support/supportsite.wss/docdisplay?lndoci d=MCGN-46AMQP&brandind=5000016 IBM System x benchmarks http://www.ibm.com/servers/eserver/xseries/benchmarks/related.html<br />
<br />
Intel Web sites Itanium 2 Hardware Developer's Manual http://www.intel.com/design/itanium2/manuals/25110901.pdf Xeon MP product overview http://www.intel.com/distributed/modules/sitelets/xeon_home.htm Hyper-Threading Technology http://www.intel.com/technology/hyperthread NetBurst Architecture http://www.intel.com/cd/ids/developer/asmo-na/eng/44004.htm<br />
<br />
1022<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Intel 64 Architecture, formerly known as Intel Extended Memory 64 Technology http://www.intel.com/technology/intel64/index.htm E7520 chipset overview http://www.intel.com/design/chipsets/E7520_E7320 E7525 Memory Controller Hub chipset overview http://www.intel.com/products/chipsets/e7525/index.htm DDR2 memory overview http://developer.intel.com/technology/memory<br />
<br />
AMD AMD64 site http://www.x86-64.org HyperTransport Consortium http://www.hypertransport.org<br />
<br />
Other hardware article: DDR vs DDRII - Fight! http://www.xbitlabs.com/articles/memory/display/ddr2.html article: Introduction to DDR-2: The DDR Memory Replacement http://www.pcstats.com/articleview.cfm?articleid=1573 Serial ATA International Organization http://www.serialata.org Auto-negotiation Valid Configuration Table http://www.cisco.com/warp/public/473/46.html#auto_neg_valid<br />
<br />
Microsoft Microsoft Windows Server 2000 TCP/IP Implementation Details http://www.microsoft.com/windows2000/techinfo/howitworks/communicati ons/networkbasics/tcpip_implement.asp Microsoft Windows Server 2003 TCP/IP Implementation Details http://www.microsoft.com/technet/prodtechnol/windowsserver2003/techn ologies/networking/tcpip03.mspx<br />
<br />
Related publications<br />
<br />
1023<br />
<br />
Performance Tuning Guidelines for Windows Server 2003 http://www.microsoft.com/windowsserver2003/evaluation/performance/tu ning.mspx Registry entry NtfsDisableLastAccessUpdate http://www.microsoft.com/resources/documentation/WindowsServ/2003/al l/deployguide/en-us/46656.asp Intfiltr utility ftp://ftp.microsoft.com/bussys/winnt/winnt-public/tools/affinity/int filtr.zip Windows Server 2003 Tools http://www.microsoft.com/windowsServer2003/downloads/tools/default.m spx WMI performance console classes http://msdn2.microsoft.com/en-us/library/aa392738.aspx KB entry "How to Overcome 4,095-MB Paging File Size Limit in Windows" http://support.microsoft.com/?kbid=237740 KB entry "MaxMpxCt and MaxCmds Limits in Windows 2000" http://support.microsoft.com/?kbid=271148 KB entry "About Cache Manager in Windows Server 2003") http://support.microsoft.com/?kbid=837331 KB entry "Description of Windows 2000 TCP Features" http://support.microsoft.com/?kbid=224829 KB entry "Windows 2000 Does Not Use Configured TCPWindowSize Registry Parameter When Accepting a Connection" http://support.microsoft.com/?kbid=263088 KB entry "TCP/IP and NBT configuration parameters for Windows 2000 or Windows NT" http://support.microsoft.com/?kbid=120642 KB entry "How to Stop the NTExecutive from Paging to Disk" http://support.microsoft.com/?kbid=184419 KB entry "Server Service Configuration and Tuning" http://support.microsoft.com/?kbid=128167<br />
<br />
1024<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
KB entry "How to Install and Use the Interrupt-Affinity Filter Tool" http://support.microsoft.com/?kbid=252867 KB entry "Terminal Server Client Connections and Logon Limited by MaxWorkItem and MaxMpxCt Values" http://support.microsoft.com/?kbid=232476 KB entry "HOW TO: Download, Install, and Remove the IIS MetaEdit 2.2 Utility" http://support.microsoft.com/?kbid=232068 Managing Memory-Mapped Files in Win32 http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dng enlib/html/msdn_manamemo.asp Windows Server 2003 Resource Kit Tools http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69-57 ff-4ae7-96ee-b18c4790cffd&displaylang=en Cache Manager in Microsoft Windows Server 2003 http://support.microsoft.com/default.aspx?scid=kb;en-us;837331 Managing Memory-Mapped Files in Win32 http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dng enlib/html/msdn_manamemo.asp Microsoft SQL Server 2005 http://www.microsoft.com/sql/2005<br />
<br />
Novell Web sites NetWare 6.5 home page http://www.novell.com/products/netware DirXML home page http://www.novell.com/products/dirxml eDirectory http://www.novell.com/products/edirectory RFC "TCP Selective Acknowledgment Options" http://www.rfc-editor.org/rfc/rfc2018.txt RFC "TCP Extensions for High Performance" http://www.rfc-editor.org/rfc/rfc1323.txt<br />
<br />
Related publications<br />
<br />
1025<br />
<br />
document "Tuning Garbage Collection with the 1.4.2 Java Virtual Machine" http://java.sun.com/docs/hotspot/gc1.4.2/ Novell Storage Services (NSS) Performance Monitoring and Tuning http://support.novell.com/techcenter/articles/ana20020701.html Enhancing TCP Performance Through the Large Window and SACK Options http://support.novell.com/techcenter/articles/ana20021203.html Link Level Load Balancing and Fault Tolerance in NetWare 6 http://support.novell.com/techcenter/articles/ana20020303.html Tuning the NetWare 6 TCP/IP Stack through SET Parameters http://support.novell.com/techcenter/articles/ana20020702.html Improving Performance by Disabling Hyper-Threading http://www.novell.com/coolsolutions/netware/features/a_hyperthread_n w.html Memory Fragmentation Issue with NetWare 6.0/6.5 http://support.novell.com/cgi-bin/search/searchtid.cgi?/10091980.htm Intel Hyper-Threading technology available on NetWare 6 http://support.novell.com/cgi-bin/search/searchtid.cgi?/10069381.htm Memory Fragmentation Issue with NetWare 6.0/6.5 http://support.novell.com/cgi-bin/search/searchtid.cgi?/10091980.htm Intel Hyper-Threading technology available on NetWare 6 http://support.novell.com/cgi-bin/search/searchtid.cgi?/10069381.htm<br />
<br />
Linux High Memory In The Linux Kernel http://kerneltrap.org/node/view/2450 Sysstat Utilities http://perso.wanadoo.fr/sebastien.godard Linux File System Hierarchy Standard http://www.pathname.com/fhs Ext3 file system http://www.redhat.com/support/wpapers/redhat/ext3<br />
<br />
1026<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Lotus Domino Lotus online Help http://notes.net/notesua.nsf Impact of Maintaining R5 Format ODS 41 Databases on Domino 6: Database Size Increases http://www.ibm.com/support/docview.wss?uid=swg21154044 NotesBench Consortium Web site http://www.notesbench.org Impact of Notes Calendaring and Scheduling (C&S) on the performance of the Lotus Domino Server ftp://ftp.lotus.com/pub/lotusweb/product/domino/dominocands.pdf Data compression in ASIC cores http://researchweb.watson.ibm.com/journal/rd/426/craft.html DB2 UDB, DB2 Connect and DB2 Information Integrator Version 8 product manuals http://www.ibm.com/software/data/db2/udb/support/manualsv8.html<br />
<br />
Oracle Oracle Technology Network http://technet.oracle.com Oracle9i Database Release 9.0.1 Documentation http://otn.oracle.com/documentation/oracle9i_arch_901.html Oracle Database 10g white papers http://otn.oracle.com/deploy/performance/WhitePapers.html Oracle9i Database Release 2 Documentation http://otn.oracle.com/documentation/oracle9i.html Oracle 10g – Oracle Database Client Installation Guide 10g Release 1 (10.1.0.2.0) for Windows (Oracle Corporation), which is available from: http://otn.oracle.com/documentation/oracle9i_arch_901.html – Database Performance with Oracle Database 10g (Oracle Corporation), which is available from: http://otn.oracle.com/deploy/performance/WhitePapers.html<br />
<br />
Related publications<br />
<br />
1027<br />
<br />
– DSS Performance in Oracle Database 10g (Oracle Corporation), which is available from: http://otn.oracle.com/deploy/performance/WhitePapers.html – Oracle Database 10g on 64-bit Linux: Ready for Enterprise-class Computing (Oracle Corporation), which is available from: http://otn.oracle.com/deploy/performance/WhitePapers.html Oracle 9i – Oracle 9i. Database Performance Guide and reference. (Oracle Corporation), which is available from: http://technet.oracle.com – Oracle 9i. Database Installation Guide. Release 2 (9.2) for Windows. (Oracle Corporation), which is available from: http://otn.oracle.com/documentation/oracle9i.html – Oracle 9i. Database Getting Started. Release 1 (9.0.1) for Windows. (Oracle Corporation), which is available from: http://otn.oracle.com/documentation/oracle9i_arch_901.html – Oracle 9i. Database Concepts. Release 1 (9.0.1). (Oracle Corporation), which is available from: http://technet.oracle.com A few papers on Oracle that are available are: – Configuring Oracle Server for VLDB by Cary V. Millsap (Oracle Corporation), which is available from: http://www.visioncg.com/whitepapers.htm http://helio.ora.com/catalog/oressentials/chapter/papers.html – The OFA Standard - Oracle 7 for Open Systems by Cary V. Millsap (Oracle Corporation), which is available from: http://www.visioncg.com/whitepapers.htm Moreover, a few papers on RAID technology describe the I/O subsystem tuning issues: – RAID: High Performance, Reliable Secondary Storage by Peter M. Chen et al. – Designing Disk Arrays for High Data Reliability by Garth A. Gibson and David A. Patterson – A Performance Evaluation of RAID Architectures by Shenze Chen and Don Towsley<br />
<br />
1028<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Citrix Citrix Home page http://www.citrix.com Citrix Knowledge Center http://support.citrix.com Citrix Web Interface home page http://www.dabcc.com/NFuse Citrix Application Delivery Infrastructure for Access On-Demand http://www.citrix.com/site/PS/products/product.asp?family=19&product ID=184 Secure Gateway for MetaFrame http://www.dabcc.com/thinsol/csg<br />
<br />
Apache Apache HTTP Server Project http://httpd.apache.org Benchmark for Web Servers http://www.mindcraft.com/Webstone use Perl to manage Apache, respond to requests for Web pages and much more http://perl.apache.org/ Compilation and installation of Apache on UNIX and UNIX-like systems http://httpd.apache.org/docs-2.0/install.html Stopping and Restarting on Unix- like systems http://httpd.apache.org/docs-2.0/stopping.html Apache HTTP Server V1.3 core features http://httpd.apache.org/docs/mod/core.html Apache core features http://httpd.apache.org/docs-2.0/mod/core.html Resolve IP - addresses to host names http://httpd.apache.org/docs-2.0/programs/logresolve.html Compress content extension http://httpd.apache.org/docs-2.0/mod/mod_deflate.html<br />
<br />
Related publications<br />
<br />
1029<br />
<br />
How to get IBM Redbooks You can order hardcopy IBM Redbooks, as well as view, download, or search for IBM Redbooks at the following Web site: ibm.com/redbooks You can also download additional materials (code samples or diskette/CD-ROM images) from this Web site.<br />
<br />
IBM Redbooks collections IBM Redbooks are also available on CD-ROMs. Click the CD-ROMs button on the IBM Redbooks Web site for information about all the CD-ROMs that are offered, as well as for information about updates and formats.<br />
<br />
1030<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Abbreviations and acronyms ABAP™<br />
<br />
Advanced Business Application Programming<br />
<br />
ASIC<br />
<br />
application-specific integrated circuit<br />
<br />
ABP<br />
<br />
address bit permuting<br />
<br />
ASP<br />
<br />
active server page<br />
<br />
ACK<br />
<br />
acknowledgment<br />
<br />
ATA<br />
<br />
AT attachment<br />
<br />
ACL<br />
<br />
access control list<br />
<br />
ATAPI<br />
<br />
ATA packet interface<br />
<br />
ACPI<br />
<br />
advanced control and power interface<br />
<br />
ATC<br />
<br />
advanced transfer cache<br />
<br />
ATM<br />
<br />
asynchronous transfer mode<br />
<br />
AD<br />
<br />
Active Directory<br />
<br />
ATS<br />
<br />
Advanced Technical Support<br />
<br />
ADSI<br />
<br />
Active Directory Services Interface<br />
<br />
AVC<br />
<br />
Access Vector Cache<br />
<br />
AWE<br />
<br />
Address Windowing Extensions<br />
<br />
AFT<br />
<br />
adapter fault tolerance<br />
<br />
AGP<br />
<br />
accelerated graphics port<br />
<br />
BASH<br />
<br />
Bourne-again shell<br />
<br />
AIDE<br />
<br />
Advanced Intrusion Detection Environment<br />
<br />
BCM<br />
<br />
BroadCom<br />
<br />
AIM<br />
<br />
application integration and middleware<br />
<br />
BDC<br />
<br />
backup domain controller<br />
<br />
BGA<br />
<br />
ball grid array<br />
<br />
ALB<br />
<br />
adaptive load balancing<br />
<br />
BIOS<br />
<br />
basic input/output system<br />
<br />
ALU<br />
<br />
arithmetic logic unit<br />
<br />
BIU<br />
<br />
bus interface unit<br />
<br />
AMB<br />
<br />
Advanced Memory Buffer<br />
<br />
BLG<br />
<br />
binary log file<br />
<br />
AMD<br />
<br />
Advanced Micro Devices, Inc.<br />
<br />
BMRS<br />
<br />
AMP<br />
<br />
Apache, MySQL, and PHP/Perl<br />
<br />
backlog management reporting system<br />
<br />
C&S<br />
<br />
calendaring and scheduling<br />
<br />
American National Standards Institute<br />
<br />
CAS<br />
<br />
column address strobe<br />
<br />
CCF<br />
<br />
cache coherency filter<br />
<br />
API<br />
<br />
application programming interface<br />
<br />
CCMS<br />
<br />
Computer Center Management System<br />
<br />
APIC<br />
<br />
advanced programmable interrupt controller<br />
<br />
CD<br />
<br />
compact disk<br />
<br />
CD-ROM<br />
<br />
APR<br />
<br />
Apache Portable Runtime<br />
<br />
compact disk read only memory<br />
<br />
ARC<br />
<br />
Advanced Risc Computing<br />
<br />
CDSL<br />
<br />
ASAP<br />
<br />
Accelerated SAP<br />
<br />
Context Dependent Symbolic Links<br />
<br />
ASCII<br />
<br />
American Standard Code for Information Interchange<br />
<br />
CEC<br />
<br />
central electronics complex<br />
<br />
CFQ<br />
<br />
Completely Fair Queuing<br />
<br />
ASF<br />
<br />
Apache Software Foundation<br />
<br />
CGI<br />
<br />
Common Gateway Interface<br />
<br />
ANSI<br />
<br />
© Copyright IBM Corp. 1998, 2000, 2002, 2004, 2007. All rights reserved.<br />
<br />
1031<br />
<br />
CI<br />
<br />
Central Instance<br />
<br />
CIFS<br />
<br />
Common Internet File System<br />
<br />
CIM<br />
<br />
DBMS<br />
<br />
data base management system<br />
<br />
Common Information Model<br />
<br />
DBWR<br />
<br />
database writer<br />
<br />
CIMOM<br />
<br />
Common Information Model Object Manager<br />
<br />
DC<br />
<br />
domain controller<br />
<br />
DDR<br />
<br />
Double Data Rate<br />
<br />
CIR<br />
<br />
committed information rate<br />
<br />
DEP<br />
<br />
Data Execution Prevention<br />
<br />
CISC<br />
<br />
complex instruction set computer<br />
<br />
DFS<br />
<br />
Distributed File System<br />
<br />
DHCP<br />
<br />
CKPT<br />
<br />
checkpoint<br />
<br />
Dynamic Host Configuration Protocol<br />
<br />
CLB<br />
<br />
Component load balancing<br />
<br />
DIA<br />
<br />
dialog<br />
<br />
CLR<br />
<br />
common language runtime<br />
<br />
DIIOP<br />
<br />
CMD<br />
<br />
command<br />
<br />
Domino Internet Inter-ORB Protocol<br />
<br />
CMOS<br />
<br />
complementary metal oxide semiconductor<br />
<br />
DIMM<br />
<br />
dual inline memory module<br />
<br />
DLL<br />
<br />
dynamic link library<br />
<br />
Capacity Manager report<br />
<br />
DLM<br />
<br />
Distributed Lock Manager<br />
<br />
DMA<br />
<br />
direct memory access<br />
<br />
Center for Microsoft Technologies<br />
<br />
DML<br />
<br />
Data Manipulation Language<br />
<br />
DMS<br />
<br />
Database Managed Space<br />
<br />
COM<br />
<br />
Component Object Model<br />
<br />
DMV<br />
<br />
dynamic management views<br />
<br />
CPI<br />
<br />
Cycles Per Instruction<br />
<br />
DMZ<br />
<br />
demilitarized zone<br />
<br />
CPU<br />
<br />
central processing unit<br />
<br />
DNS<br />
<br />
domain name system<br />
<br />
CRC<br />
<br />
cyclic redundancy check<br />
<br />
DOS<br />
<br />
disk operating system<br />
<br />
CRM<br />
<br />
Customer Relationship Management<br />
<br />
DP<br />
<br />
dual processor<br />
<br />
CSG<br />
<br />
chip select group<br />
<br />
DRAM<br />
<br />
dynamic random access memory<br />
<br />
CSMA<br />
<br />
carrier-sense multiple access<br />
<br />
DRBD<br />
<br />
CSNW<br />
<br />
Client Services for NetWare<br />
<br />
Distributed Replicated Block Device<br />
<br />
CSRAM<br />
<br />
custom static RAM<br />
<br />
DRS<br />
<br />
CSS<br />
<br />
Cascading Style Sheets<br />
<br />
Distributed Resource Scheduler<br />
<br />
CSV<br />
<br />
comma separated variable<br />
<br />
DSA<br />
<br />
Directory Store Access<br />
<br />
DAC<br />
<br />
dual address cycle<br />
<br />
DSO<br />
<br />
Dynamic Shared Object<br />
<br />
DAS<br />
<br />
Direct Attached Storage<br />
<br />
DSS<br />
<br />
decision support system<br />
<br />
DB<br />
<br />
database<br />
<br />
DTA<br />
<br />
Database Tuning Advisor<br />
<br />
DBA<br />
<br />
database administrator<br />
<br />
DW<br />
<br />
data warehousing<br />
<br />
DBCC<br />
<br />
database consistency checker<br />
<br />
EB<br />
<br />
Exabytes<br />
<br />
DBIID<br />
<br />
database instance ID<br />
<br />
ECB<br />
<br />
event control block<br />
<br />
ECC<br />
<br />
error checking and correcting<br />
<br />
CMR CMT CMT<br />
<br />
1032<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
EDO<br />
<br />
extended data out<br />
<br />
FTP<br />
<br />
file transfer protocol<br />
<br />
EFREI<br />
<br />
Ecole d'ingénieur informatique<br />
<br />
Gb<br />
<br />
gigabit<br />
<br />
GB<br />
<br />
gigabyte<br />
<br />
EIDE<br />
<br />
enhanced IDE<br />
<br />
GCC<br />
<br />
GNU Compiler Collection<br />
<br />
EISA<br />
<br />
extended ISA<br />
<br />
GDB<br />
<br />
GNU Project Debugger<br />
<br />
EL<br />
<br />
execution layer<br />
<br />
GDI<br />
<br />
graphics device interface<br />
<br />
EMEA<br />
<br />
Europe, Middle East, Africa<br />
<br />
GEC<br />
<br />
Gigabit EtherChannel<br />
<br />
ENET<br />
<br />
Ethernet<br />
<br />
GIF<br />
<br />
graphic interchange format<br />
<br />
EPIC<br />
<br />
explicitly parallel instruction computing<br />
<br />
GNU<br />
<br />
GNU’s Not Unix<br />
<br />
GPR<br />
<br />
general purpose register<br />
<br />
GRE<br />
<br />
Generic Routing Encapsulation<br />
<br />
GRUB<br />
<br />
Grand Unified Bootloader<br />
<br />
GUI<br />
<br />
graphical user interface<br />
<br />
HA<br />
<br />
high availability<br />
<br />
HAL<br />
<br />
hardware abstraction layer<br />
<br />
ERP<br />
<br />
enterprise resource planning<br />
<br />
ESCON<br />
<br />
enterprise systems connection<br />
<br />
ESE<br />
<br />
Extensible Storage Engine<br />
<br />
EVMS<br />
<br />
Enterprise Volume Management System<br />
<br />
EXA<br />
<br />
Enterprise X-Architecture<br />
<br />
EXP<br />
<br />
expansion<br />
<br />
HAM<br />
<br />
hot-add memory<br />
<br />
F/W<br />
<br />
fast/wide<br />
<br />
HBA<br />
<br />
host bus adapter<br />
<br />
FAMM<br />
<br />
Full Array Memory Mirroring<br />
<br />
HCA<br />
<br />
host channel adapter<br />
<br />
FAQ<br />
<br />
frequently asked questions<br />
<br />
HCL<br />
<br />
hardware compatibility list<br />
<br />
FAStT<br />
<br />
Fibre Array Storage Technology<br />
<br />
HCT<br />
<br />
hardware compatibility testing<br />
<br />
HE<br />
<br />
high end<br />
<br />
FAT<br />
<br />
file allocation table<br />
<br />
HPC<br />
<br />
high performance computing<br />
<br />
FBD<br />
<br />
Fully Buffered DIMMs<br />
<br />
HPMA<br />
<br />
FB-DIMM<br />
<br />
Fully Buffered DIMMs<br />
<br />
High Performance Memory Array<br />
<br />
FC<br />
<br />
Fibre Channel<br />
<br />
HT<br />
<br />
Hyper-Threading<br />
<br />
FC-AL<br />
<br />
Fibre Channel-arbitrated loop<br />
<br />
HTML<br />
<br />
Hypertext Markup Language<br />
<br />
FDDI<br />
<br />
fiber distributed data interface<br />
<br />
HTTP<br />
<br />
FEC<br />
<br />
Fast EtherChannel<br />
<br />
Hypertext Transmission Protocol<br />
<br />
FIFO<br />
<br />
first in first out<br />
<br />
I/O<br />
<br />
input/output<br />
<br />
FP<br />
<br />
floating point<br />
<br />
I/OAT<br />
<br />
I/O Acceleration Technology<br />
<br />
FPM<br />
<br />
fast page memory<br />
<br />
IAM<br />
<br />
index allocation map<br />
<br />
FQDN<br />
<br />
fully qualified domain name<br />
<br />
IBM<br />
<br />
International Business Machines Corporation<br />
<br />
FRS<br />
<br />
File Replication service<br />
<br />
ICA<br />
<br />
FS<br />
<br />
fast skinny<br />
<br />
Independent Computing Architecture<br />
<br />
FSB<br />
<br />
front-side bus<br />
<br />
Abbreviations and acronyms<br />
<br />
1033<br />
<br />
ICMP<br />
<br />
Internet control message protocol<br />
<br />
JS<br />
<br />
JavaScript™<br />
<br />
JVM™<br />
<br />
Java Virtual Machine<br />
<br />
ID<br />
<br />
identifier<br />
<br />
KB<br />
<br />
kilobyte<br />
<br />
IDE<br />
<br />
integrated drive electronics<br />
<br />
KCC<br />
<br />
IEEE<br />
<br />
Institute of Electrical and Electronics Engineers<br />
<br />
Knowledge Consistency Checker<br />
<br />
KDE<br />
<br />
K Desktop Environment<br />
<br />
IIOP<br />
<br />
Internet InterORB Protocol<br />
<br />
LACP<br />
<br />
IIS<br />
<br />
Internet Information Services<br />
<br />
Link Aggregation Control Protocol<br />
<br />
IMAP<br />
<br />
Internet Mail Access Protocol<br />
<br />
LAN<br />
<br />
local area network<br />
<br />
IMC<br />
<br />
Internet Mail Connector<br />
<br />
LBA<br />
<br />
Logical Block Address<br />
<br />
IMS™<br />
<br />
Internet mail service<br />
<br />
LDAP<br />
<br />
IOAT<br />
<br />
I/O Acceleration Technology<br />
<br />
Lightweight Directory Access Protocol<br />
<br />
IOQ<br />
<br />
in-order queue<br />
<br />
LE<br />
<br />
low end<br />
<br />
IP<br />
<br />
Internet Protocol<br />
<br />
LGWR<br />
<br />
Log Writer<br />
<br />
IPF<br />
<br />
Itanium Processor Family<br />
<br />
LME<br />
<br />
large memory enabled<br />
<br />
IPI<br />
<br />
intelligent peripheral interface<br />
<br />
LMS<br />
<br />
Lock Management Server<br />
<br />
IPMI<br />
<br />
Intelligent Platform Management Interface<br />
<br />
LN<br />
<br />
logical node<br />
<br />
LOB<br />
<br />
large object<br />
<br />
IPSEC<br />
<br />
IP Security<br />
<br />
LPC<br />
<br />
Local Procedure Call<br />
<br />
IPX<br />
<br />
Internetwork packet exchange<br />
<br />
LPT<br />
<br />
line printer<br />
<br />
IRDY<br />
<br />
initiator ready<br />
<br />
LRU<br />
<br />
least recently used<br />
<br />
IRQ<br />
<br />
interrupt request<br />
<br />
LSA<br />
<br />
Local Security Authority<br />
<br />
IS<br />
<br />
information store<br />
<br />
LSL<br />
<br />
Link Support Layer.<br />
<br />
ISA<br />
<br />
industry standard architecture<br />
<br />
LSM<br />
<br />
Linux Security Modules<br />
<br />
ISAPI<br />
<br />
Internet Services application programming interface<br />
<br />
LTC<br />
<br />
Linux Technology Center<br />
<br />
LUN<br />
<br />
logical unit number<br />
<br />
ISBN<br />
<br />
international standard book number<br />
<br />
LV<br />
<br />
low voltage<br />
<br />
ISDN<br />
<br />
Integrated Services Digital Network<br />
<br />
LVD<br />
<br />
Low Voltage Differential<br />
<br />
LVM<br />
<br />
Logical Volume Manager<br />
<br />
MAC<br />
<br />
memory address controller; media access control<br />
<br />
MADP<br />
<br />
memory address data path<br />
<br />
MAN<br />
<br />
metropolitan area network<br />
<br />
MAPI<br />
<br />
messaging application programming interface<br />
<br />
Mb<br />
<br />
megabit<br />
<br />
MB<br />
<br />
megabyte<br />
<br />
ISI<br />
<br />
Inter-symbol interference<br />
<br />
IT<br />
<br />
information technology<br />
<br />
ITS<br />
<br />
Internet Transaction Server<br />
<br />
ITSO<br />
<br />
International Technical Support Organization<br />
<br />
JDB<br />
<br />
Java Debugger<br />
<br />
JPEG<br />
<br />
Joint Photographic Experts Group<br />
<br />
1034<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
MCSE<br />
<br />
Microsoft Certified Systems Engineer<br />
<br />
MXT<br />
<br />
Memory eXpansion Technology<br />
<br />
MDAC<br />
<br />
Microsoft Data Access Components<br />
<br />
NAS<br />
<br />
network addressable storage<br />
<br />
NCP<br />
<br />
Novell Core Protocol<br />
<br />
MDDB<br />
<br />
multidimensional databases<br />
<br />
NCQ<br />
<br />
Native Command Queuing<br />
<br />
MESI<br />
<br />
modified exclusive shared invalid<br />
<br />
NDIS<br />
<br />
network driver interface specification<br />
<br />
MHz<br />
<br />
megahertz<br />
<br />
NDS<br />
<br />
NetWare Directory Services<br />
<br />
MIOC<br />
<br />
Memory and I/O Bridge Controller<br />
<br />
NES<br />
<br />
NetWare Enterprise Web Server<br />
<br />
MIS<br />
<br />
management information system<br />
<br />
NFS<br />
<br />
network file system<br />
<br />
MLT<br />
<br />
Master Latency Timer<br />
<br />
NIC<br />
<br />
network interface card<br />
<br />
MMC<br />
<br />
Microsoft Management Console<br />
<br />
NIF<br />
<br />
Notes Index Facility<br />
<br />
NLBS<br />
<br />
MMX™<br />
<br />
multimedia extensions<br />
<br />
Network Load Balancing Services<br />
<br />
MOESI<br />
<br />
modified owner exclusive shared invalid<br />
<br />
NLM<br />
<br />
NetWare loadable module<br />
<br />
NMI<br />
<br />
non-maskable interrupt NetNews transfer protocol<br />
<br />
MP<br />
<br />
multiprocessor<br />
<br />
NNTP<br />
<br />
MPEG<br />
<br />
Moving Pictures Experts Group<br />
<br />
NOS<br />
<br />
network operating system<br />
<br />
NRM<br />
<br />
NetWare Remote Manager<br />
<br />
MPK<br />
<br />
multiprocessor kernel<br />
<br />
NRPC<br />
<br />
Notes Remote Procedure Call<br />
<br />
MSCS<br />
<br />
Microsoft Cluster Server<br />
<br />
NSF<br />
<br />
Notes Storage File<br />
<br />
MSDE<br />
<br />
Microsoft SQL Server Desktop Engine<br />
<br />
NSS<br />
<br />
Novell Storage Services<br />
<br />
NTFS<br />
<br />
NT file system<br />
<br />
MSDN<br />
<br />
Microsoft Developer Network<br />
<br />
NTP<br />
<br />
Network Time Protocol<br />
<br />
MSI<br />
<br />
Microsoft Installer<br />
<br />
NUMA<br />
<br />
Non-Uniform Memory Access<br />
<br />
MSIE<br />
<br />
Microsoft Internet Explorer<br />
<br />
ODS<br />
<br />
on-disk structure<br />
<br />
MSR<br />
<br />
Machine Status Register<br />
<br />
OFA<br />
<br />
Oracle Financial Analyzer<br />
<br />
MSS<br />
<br />
maximum segment size<br />
<br />
OLAP<br />
<br />
online analytical processing<br />
<br />
MTA<br />
<br />
mail transport agent<br />
<br />
OLTP<br />
<br />
online transaction processing<br />
<br />
MTTDL<br />
<br />
mean time to data loss<br />
<br />
OOB<br />
<br />
Out of band<br />
<br />
MTTF<br />
<br />
mean time to failure<br />
<br />
ORB<br />
<br />
Object Request Broker<br />
<br />
MTTR<br />
<br />
mean time to repair<br />
<br />
OS<br />
<br />
operating system<br />
<br />
MTU<br />
<br />
maximum transfer unit<br />
<br />
OSI<br />
<br />
Open Systems Interconnect<br />
<br />
MUI<br />
<br />
Management User Interface<br />
<br />
OSS<br />
<br />
online service system<br />
<br />
MUX<br />
<br />
multiplexer<br />
<br />
OWA<br />
<br />
Outlook® Web Access<br />
<br />
PAE<br />
<br />
Physical Address Extension<br />
<br />
Abbreviations and acronyms<br />
<br />
1035<br />
<br />
PATA<br />
<br />
Parallel ATA<br />
<br />
RAC<br />
<br />
Real Application Clusters<br />
<br />
PC<br />
<br />
personal computer<br />
<br />
RAID<br />
<br />
PCB<br />
<br />
printed circuit board<br />
<br />
redundant array of independent disks<br />
<br />
PCI<br />
<br />
peripheral component interconnect<br />
<br />
RAID-M<br />
<br />
PCI-E<br />
<br />
PCI Express<br />
<br />
redundant array of inexpensive DRAMs for memory<br />
<br />
PCPU<br />
<br />
physical CPU<br />
<br />
RAM<br />
<br />
random access memory<br />
<br />
PCRE<br />
<br />
Perl Compatible Regular Expressions<br />
<br />
RAS<br />
<br />
remote access services; row address strobe<br />
<br />
PDC<br />
<br />
primary domain controller<br />
<br />
RBS<br />
<br />
redundant bit steering<br />
<br />
PDO<br />
<br />
PHP Data Objects<br />
<br />
RCG<br />
<br />
RAS/CAS Generator<br />
<br />
PEAR<br />
<br />
PHP Extension and Application Repository<br />
<br />
RDBMS<br />
<br />
relational database management system<br />
<br />
PFS<br />
<br />
page free space<br />
<br />
RDC<br />
<br />
Remote Desktop Client<br />
<br />
PGA<br />
<br />
program global area; pin grid array<br />
<br />
RDIMM<br />
<br />
registered DIMM<br />
<br />
RDMA<br />
<br />
PID<br />
<br />
process ID<br />
<br />
Remote Direct Memory Access<br />
<br />
PMTU<br />
<br />
Path Maximum Transmission Unit<br />
<br />
RDP<br />
<br />
Remote Desktop Protocol<br />
<br />
RECO<br />
<br />
recovery<br />
<br />
PNG<br />
<br />
Portable Network Graphics<br />
<br />
RFC<br />
<br />
request for comments<br />
<br />
POSIX<br />
<br />
Portable Operating System Interface<br />
<br />
RHEL<br />
<br />
Red Hat Enterprise Linux<br />
<br />
RISC<br />
<br />
reduced instruction set computer<br />
<br />
RMAP-VM<br />
<br />
reverse mapping-virtual memory<br />
<br />
RMI<br />
<br />
Remote Method Invocation<br />
<br />
RODC<br />
<br />
Read-only domain controller<br />
<br />
ROLAP<br />
<br />
Relational online analytical processing<br />
<br />
POST<br />
<br />
power on self test<br />
<br />
PP<br />
<br />
production planning<br />
<br />
PRI<br />
<br />
priority<br />
<br />
PS<br />
<br />
Personal System<br />
<br />
PSA<br />
<br />
POSIX semantic agent<br />
<br />
PSE<br />
<br />
Page Size Extension<br />
<br />
PSM<br />
<br />
Platform Support Module<br />
<br />
RPC<br />
<br />
remote procedure call<br />
<br />
PTE<br />
<br />
Page Table Entry<br />
<br />
RPM<br />
<br />
revolutions per minute<br />
<br />
PXB<br />
<br />
PCI Expander Bridge<br />
<br />
RPO<br />
<br />
PXE<br />
<br />
Preboot Execution Environment<br />
<br />
rotational positioning optimization<br />
<br />
RSS<br />
<br />
Receive-side scaling<br />
<br />
QDR<br />
<br />
quad data rate<br />
<br />
RT<br />
<br />
real time<br />
<br />
QoS<br />
<br />
Quality of Service<br />
<br />
RTM<br />
<br />
release to manufacturing<br />
<br />
R/W<br />
<br />
read/write<br />
<br />
RTO<br />
<br />
retransmission time-out<br />
<br />
RA<br />
<br />
read ahead<br />
<br />
RTP<br />
<br />
Research Triangle Park<br />
<br />
1036<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
RTT<br />
<br />
round trip time<br />
<br />
SNP<br />
<br />
Scalable Networking Pack<br />
<br />
RXE<br />
<br />
Remote Expansion Enclosure<br />
<br />
SPI<br />
<br />
SCSI-3 parallel interface<br />
<br />
SAC<br />
<br />
System Address Component<br />
<br />
SPO<br />
<br />
spool<br />
<br />
SACK<br />
<br />
Selective Acknowledgment<br />
<br />
SPX<br />
<br />
sequenced packet exchange<br />
<br />
SAM<br />
<br />
Security Accounts Manager<br />
<br />
SQL<br />
<br />
structured query language<br />
<br />
SAN<br />
<br />
storage area network<br />
<br />
SRAM<br />
<br />
static RAM<br />
<br />
SAPS<br />
<br />
SAP Application Benchmark Performance Standard<br />
<br />
SRAT<br />
<br />
Static Resource Allocation Table<br />
<br />
SAS<br />
<br />
Serial Attached SCSI<br />
<br />
SSA<br />
<br />
serial storage architecture<br />
<br />
SATA<br />
<br />
Serial ATA<br />
<br />
SSE<br />
<br />
Streaming SIMD Extensions<br />
<br />
SBCCS<br />
<br />
single byte command code set<br />
<br />
SSH<br />
<br />
Secure Shell<br />
<br />
SCSI<br />
<br />
small computer system interface<br />
<br />
SSL<br />
<br />
Secure Sockets Layer<br />
<br />
SSP<br />
<br />
Serial SCSI Protocol<br />
<br />
SCT<br />
<br />
system compatibility testing<br />
<br />
STP<br />
<br />
SATA Tunneled Protocol<br />
<br />
SD<br />
<br />
sales and distribution<br />
<br />
SUMO<br />
<br />
SDBU<br />
<br />
SD benchmark users<br />
<br />
Sufficiently Uniform Memory Organization<br />
<br />
SDC<br />
<br />
System Data Component<br />
<br />
SWAT<br />
<br />
SDNU<br />
<br />
SD normalized users<br />
<br />
Samba Web Administration Tool<br />
<br />
SDRAM<br />
<br />
static dynamic RAM<br />
<br />
TB<br />
<br />
terabyte<br />
<br />
SDRF<br />
<br />
SD Reference Factor<br />
<br />
TCB<br />
<br />
Transport Control Block<br />
<br />
SEC<br />
<br />
single edge connector<br />
<br />
TCO<br />
<br />
total cost of ownership<br />
<br />
SGA<br />
<br />
system global area<br />
<br />
TCP<br />
<br />
Transmission Control Protocol<br />
<br />
SID<br />
<br />
system ID<br />
<br />
TCP/IP<br />
<br />
SIMD<br />
<br />
single instruction multiple data<br />
<br />
Transmission Control Protocol/Internet Protocol<br />
<br />
SLES<br />
<br />
SUSE Linux Enterprise Server<br />
<br />
TCQ<br />
<br />
tagged command queuing<br />
<br />
TFS<br />
<br />
traditional file system<br />
<br />
SLT<br />
<br />
slot<br />
<br />
TID<br />
<br />
SMB<br />
<br />
server message block<br />
<br />
Technical Information Document<br />
<br />
SMI<br />
<br />
synchronous memory interface<br />
<br />
TLB<br />
<br />
translation lookaside buffer<br />
<br />
TLS<br />
<br />
Transport Layer Security<br />
<br />
SMON<br />
<br />
System Monitor<br />
<br />
TOE<br />
<br />
TCP offload engine<br />
<br />
SMP<br />
<br />
symmetric multiprocessors<br />
<br />
TPC<br />
<br />
SMS<br />
<br />
System Managed Space<br />
<br />
Transaction Processing Council<br />
<br />
SMTP<br />
<br />
simple mail transfer protocol<br />
<br />
TRDY<br />
<br />
target ready<br />
<br />
SNA<br />
<br />
systems network architecture<br />
<br />
TSV<br />
<br />
tab separated variable<br />
<br />
SNMP<br />
<br />
simple network management protocol<br />
<br />
TTY<br />
<br />
teletypewriter<br />
<br />
TXT<br />
<br />
text<br />
<br />
Abbreviations and acronyms<br />
<br />
1037<br />
<br />
UBM<br />
<br />
Unified Buffer Manager<br />
<br />
UDB<br />
<br />
DB2 Universal Database™<br />
<br />
UDP UHCI<br />
<br />
WMI<br />
<br />
Windows Management Instrumentation<br />
<br />
user datagram protocol<br />
<br />
WSRM<br />
<br />
Universal Host Controller Interface<br />
<br />
Windows Systems Resource Manager<br />
<br />
WT<br />
<br />
write through (cache)<br />
<br />
UM<br />
<br />
universal manageability<br />
<br />
WTS<br />
<br />
Windows Terminal Server<br />
<br />
UML<br />
<br />
Unified Modeling Language<br />
<br />
WWW<br />
<br />
World Wide Web<br />
<br />
UMS<br />
<br />
User Model Schedulers<br />
<br />
XML<br />
<br />
Extensible Markup Language<br />
<br />
UNC<br />
<br />
universal naming convention<br />
<br />
XOR<br />
<br />
exclusive or<br />
<br />
URI<br />
<br />
Universal Resource Identifier<br />
<br />
ZIF<br />
<br />
zero insertion force<br />
<br />
URL<br />
<br />
universal resource locator<br />
<br />
USB<br />
<br />
universal serial bus<br />
<br />
UTP<br />
<br />
unshielded twisted pair<br />
<br />
VC<br />
<br />
VirtualCenter<br />
<br />
VIN<br />
<br />
Virtual Infrastructure Node<br />
<br />
VLAN<br />
<br />
Virtual LAN<br />
<br />
VLDB<br />
<br />
very large database<br />
<br />
VLIW<br />
<br />
very large instruction word<br />
<br />
VM<br />
<br />
virtual machine<br />
<br />
VMCB<br />
<br />
virtual machine control block<br />
<br />
VMCS<br />
<br />
virtual-machine control structure<br />
<br />
VMFS<br />
<br />
virtual machine file system<br />
<br />
VMM<br />
<br />
Virtual Machine Manager, virtual memory manager<br />
<br />
VMX<br />
<br />
Virtual Machine Extensions<br />
<br />
VPN<br />
<br />
virtual private network<br />
<br />
VRM<br />
<br />
voltage regulator module<br />
<br />
VT<br />
<br />
Virtualization Technology<br />
<br />
WA<br />
<br />
Washington<br />
<br />
WAN<br />
<br />
wide area network<br />
<br />
WB<br />
<br />
write back (cache)<br />
<br />
WBEM<br />
<br />
Web-based Enterprise Management<br />
<br />
WCAT<br />
<br />
Web Capacity Analysis Tool<br />
<br />
WINS<br />
<br />
Windows Internet Naming Service<br />
<br />
1038<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Index Symbols /3GB parameter in BOOT.INI 151, 334, 337 /PAE parameter 336 /proc 382<br />
<br />
Numerics 10 Gigabit Ethernet 256 4 GB, more than 151 64-bit 64 Linux 389 Windows 335 40K7547, Passthru card for x3755 57 64-bit Opteron 64-bit computing 61–66 64-bit mode 63 addressable memory 66 applications 64 benefits 64 definition 61 Intel 64 Technology 61 Intel processors 43 8.3 filenames 359<br />
<br />
A abstract xxiii actions 651 disk Linux 702 Windows 670 memory Linux 697 Windows 664 network Linux 704 Windows 678 processor Linux 693 Windows 660 Active Directory 14 Active Memory 33 adapter keying 89<br />
<br />
adding drives 202 Adjacent Sector Prefetch 74 advanced ECC memory 156 Advanced Memory Buffer 137, 139 Advanced Transfer Cache 42, 44 AMD See also Opteron AMD64 53, 62 AMD64 modes 62 HyperTransport 123–124, 126 Pacifica 54, 83, 412 analyzing performance 633 Linux 687–705 Windows 655–685 application scalability 108 applications Citrix MetaFrame 951–980 DB2 847–859 file server 745 Lotus Domino 763–830 Microsoft IIS 981–1016 Oracle 923–949 print server 761 SQL Server 861 VMware ESX Server 425 Windows Terminal Services 951–980 associativity, cache 67 ATM 236 auditing 362 authentication servers 13 AWE 154 SQL Server 906<br />
<br />
B background information 631 backup software 363 bandwidth, memory 144 baseline measurements 4, 630 bcopies 251 bdflush 391 benchmark STREAM 147 System x Performance Lab 5<br />
<br />
Index<br />
<br />
1039<br />
<br />
benchmarks 6 big endian 60 binary translation 81 binding order, network protocols 682 BIOS memory configuration 161 Blackford 114 BladeCenter 12 block diagrams NUMA (x3950) 102 Opteron 126 x3755 56 BOOT.INI /3GB parameter 151, 334, 337 /PAE parameter 336 bottlenecks actions 651 disk Linux 702 Windows 670 memory Linux 697 Windows 664 network Linux 704 Windows 678 processor Linux 693 Windows 660 Capacity Manager 620 CPU subsystem Linux 692 Windows 656 determining 629 disk subsystem 667 ESX Server 579–589 latent 635 Linux 687–705 memory subsystem 661 network subsystem 237, 672 Windows 655–685 worksheets 634 bridges, PCI 94 buffered SDRAMs 134 busmaster devices 85<br />
<br />
C cables, SAS 176<br />
<br />
1040<br />
<br />
cache Advanced Transfer Cache 42 associativity 67 cache coherency filter 108 cache line 144 disk 639 effect on performance 71 RAID adapter 216 write-back versus write-through 214 XceL4v 118 Capacity Manager 591–623 CMR files 593 concept 592 creating a report definition 598 generating a report 605 IBM Director management console 595 installation 594 Linux 690 Monitor Activator 596 new report definition 598 performance analysis 617 bottlenecks 620 critical threshold 620 files created 619 forecast 622 forecasting bottlenecks 620 HTML 618 icons 618 latent bottleneck 620 monitor settings 621 recommendations 617 reports 617 thresholds 620 warning threshold 620 what it does 617 Performance console, compared with 472 predefined report definitions 597 REALTIME.SLT 593 report generator 597 duration 598 global sampling frequency 603 minimum and maximum values 599 monitors, selecting 602 output method 600 report file names 602 reports, predefined 604 sampling 599, 603 timeout 604 times 600<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
report viewer 607 column titles 612 graph pane 613 hypergraph view 612 icon size 612 icon view 612 minimum and maximum values 615 monitor pane 613 selecting systems 613 sorting systems 611 system pane 611 table view 611 thresholds 609 trend graph 616 saving to file 601 scheduling reports 601, 606 SLT files 593 TREND.SLT 593 TXT files 593 when to use 473 CAS 145 case studies 707 Center for Microsoft Technologies, IBM 7 checksum offload 259 Chimney Offload 273 Chipkill 156 chipsets 97–127 AMD 123 cache coherency filter 108 design 100 eight-way configurations 106 HyperTransport 124 Intel 5000 family 114 Intel E7520 112 Intel E8500 121 latency 102 MESI 105 MOESI 106 NUMA 101 overview 98 performance 100 scalability 100 ServerWorks Grand Champion 4.0 HE/LE 111 snoop cycles 105 SRAT table 103 XA-64e 116 XceL4v 118 chkconfig command 374 Citrix MetaFrame 951–980<br />
<br />
client apps, unused features 973 clients 971 ICA 970 performance 953 protocols 970 published applications 971 RDC 970 scalability 953 scale up/out 954 Citrix Presentation Server scale-out 12 clock speed 36, 72 Clovertown 49 CMR files 593 communication servers 21 compatibility mode 63 compression Domino 819, 822 Microsoft IIS 1002 NTFS 360, 758 concepts 187 Consolidated Backup 460 Core microarchitecture 50 cores, processor 36, 45 CPU subsystem 35, 45–78 64-bit computing, benefits of 64 64-bit mode 63 adding processors 660 Adjacent Sector Prefetch 74 Advanced Digital Media Boost 52 Advanced Smart Cache 51 Advanced Transfer Cache 42, 44 affinity DB2 850 Linux 693 SQL Server 887 Windows 330 Windows Terminal Services 967 AMD64 53, 62 analysis example 730, 737 big endian 60 bottlenecks 647 Linux 693 Windows 656 cache 32, 71 cache associativity 67 cache coherency filter 108 case study 737 clock speed 36, 72<br />
<br />
Index<br />
<br />
1041<br />
<br />
Clovertown 49 comparison 67 compatibility mode 63 Core microarchitecture 50 cores 36, 45 Demand-Based Switching 49 Dempsey 47 design 100 dual core 45 eight-way configurations 106 EM64T 62 encryption 64 endian 60 EPIC 60 execution pipeline 37 floating-point operations 37 Foster 42 Hardware Prefetcher 74 hardware scalability 100 history 36 Hyper-Threading 39 HyperTransport 123, 126 HyperTransport links 56 IA-32e mode 63 IA64 60 Intel 64 Technology 62 Intelligent Power Capability 51 introduction 32 Irwindale 43 Itanium 2 59 latency 102 legacy mode 62 Linux bottlenecks 692 tuning 388 logical processors 39 long mode 63 memory addressing 66 memory, affect on 110 MESI 105 MOESI 106 NetBurst architecture 36 network, effect on 251 Nocona 43 NUMA 101, 149 Opteron 52 Passthru card for x3755 56 Paxville 46 performance 66, 100<br />
<br />
1042<br />
<br />
performance analysis 647 pipeline 37 Potomac 44 prefetch 74 Prestonia 42 primary counters 638 quad pumped 38 queue length 659 replace processors 660 scaling 72 shared L3 cache 47 SIMD 38 single core 36 Sledgehammer 52 Smart Memory Access 52 SMP defined 101 linear improvement 109 Linux 389 type of server 109 SMP scaling 72 snoop cycles 105 software scalability 108 Static Resource Allocation Table 103 technology 36 Tigerton 50 TOE 261 Tulsa 47 tuning options 660 upgrade processors 660 Virtualization Technology 46 Wide Dynamic Execution 50 Woodcrest 48 worksheet 647 Xeon DP 42 Xeon MP 43 Cranford 45 Cycles Per Instruction 98<br />
<br />
D Data Execution Prevention 153, 301 database servers 17 DB2 847–859 Oracle 923–949 SQL Server 861 datagrams 237 DB2 847–859 affinity 850<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
buffer pool 853 Buffer Pool Hit Ratio (%) 855 buffer pools 854 caching of DB2 data files 854 counters Buffer Pool Hit Ratio (%) 855 Package Cache Hit Ratio (%) 851 Physical Disk Queue Length 858 Processor, %Processor Time 851 Total Synchronous I/O Time (ms) 858 CPU subsystem 849 Database Managed Space 856 DB2NTNOCACHE setting 854 DEFAULT_DEGREE setting 849 disk subsystem 855 extent 857 Hyper-Threading 850 logical nodes 849 memory subsystem 851 networking subsystem 858 NEWLOGPATH parameter 856 nodes 849 NUM_IOSERVERS setting 856 Package Cache Hit Ratio (%) 851 page size 857 parallel I/O 855 parameter markers 851 performance 848 Performance Wizard 852 Physical Disk Queue Length 858 prefetch size 857 prefetching 855 processor affinity 850 processor subsystem 849 Processor, %Processor Time 851 processors, number of 73 RQRIOBLK parameter 859 RUNSTATS command 858 segmenting data 853 server type 17 shared-nothing cluster 849 sort heap 854 System Managed Space 856 table spaces 853, 856 Total Synchronous I/O Time (ms) 858 Windows tuning 848 DDR memory 135 DDR2 135 AMD Opteron support 54<br />
<br />
performance 136 X3 Architecture 160 Demand-Based Switching 49 Dempsey 47 device drivers 7, 219 DHCP servers 24 DIMMs 131, 160 disk partition sizes 445 disk subsystem 169–229 See also DS4000 See also Fibre Channel See also RAID levels See also SAS See also SCSI See also Serial ATA See also ServeRAID access time calculations 640 active data set size 203 adapter cache size 216 adding drives 202, 670, 702 analysis example 733, 738 bottlenecks 639 cables 176 cache size 216 calculations 640 capacity 202 case study 738 command overhead 205 common bottlenecks 641 comparison SAS-SCSI 177 data set size 203 data transfer rate 170 device drivers 219 DISKPERF 668 drive performance 205 drives, number of 202 EIDE 170, 178 Ext3 file system 397 Fibre Channel 186, 220 firmware 220 fragmentation 361 I/O operation 640 interface data rate 205 interleave depth 207 introduction 33 iSCSI 171, 189 large arrays 199 latency 170, 734 Linux 395<br />
<br />
Index<br />
<br />
1043<br />
<br />
Linux bottlenecks 698 logical drive configurations 206 Low Voltage Differential 171 media data rate 205 multiple SCSI buses 214 NAS 185 operation 172 optimization 642 paging 666 Parallel ATA 178 performance analysis 639 performance factors 200 platters 170 primary counters 638 protocols 175 RAID levels 191 RAID rebuild time 217 RAID strategy 201, 642 rebuild time 217 ReiserFS file system 398 relative speed 169 rotational latency 173, 205 rotational positioning optimization 206, 642 rules of thumb 228, 643 SAN 183 SAS 170, 174 SATA 170, 176 SCSI bus speed 213 seek operation 172 seek time 170, 205, 641 sequential I/O 641 Serial ATA 178 serial technology 171 servo track 172 spread of data 203 SSA 170 steps 172 stripe size affect on performance 642 case study 733, 739 concept 207 Fibre Channel 233 Linux 402 stroke 203 tuning options 670 upgrading, effect of 748 Windows bottlenecks 667 worksheet 639 write-back versus write-through 214<br />
<br />
1044<br />
<br />
DISKPERF command 212, 502 Distributed Resource Scheduler 460 DMA devices 85 DMA transfers 237, 240 dmesg command 540 DNS servers 24 domain controllers 13–14 Domino See Lotus Domino double-ranked DIMMs 133 downloads 100 DRAM chips 131 DS4000 187 See also Fibre Channel DS4000 Storage Manager 229 iSCSI, compare with 189 SATA comparison 226 SCSI protocol 223 throughput 221 two controllers 225 dual core 45 dynamic Web pages 20<br />
<br />
E ECC memory 131, 156 EIDE 170, 178 EM64T See Intel 64 Technology e-mail server 18 encryption 362 endian 60 EPIC 60 ERP See also SAP R/3 case study 737 ESX Server 425–468 See also ESX Server 2.5 See also ESX Server 3 /proc file system 447 aggregation 437 Apache memory used 450 authentication timeout 451 auto-negotiate 436 bottlenecks 428 BusLogic SCSI driver 440 channelling 429 concepts 426 connectionSetupTimeout 451<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Console OS tuning 448 CPU configurations 427 CPU overhead 429 devices, unused 440 disk subsystem 429 duplex 436 esxtop 580–587 external storage 435 Fibre Channel 429 front-side bus bandwidth 428 Gigabit Ethernet 429 home node 430 how it works 426 http daemon 449 Hyper-Threading 41, 427, 430 IBM Director Agent 449 IDE drives 439 important subsystems 428 initial placement 431 kernel tuning 435 LACP 429, 437 link aggregation 437 load balancing 431 logical processors 427 LSI Logic SCSI driver 440 measuring performance 579 memhard 451 memory 428 Memory Affinity option 432 memory allocation 439 mingetty 449 moving memory 431 network driver 441 network link aggregation 437 network speed 436 NIC configuration 463 NUMA support 430, 433 Only Use Processors option 432 overhead 429 Page Migration Rate option 432 page sharing 435 partitioning 434 performance measurement 579 prefetching 428 proc file system 447 RAID-1, use of 428 renice 450 SCSI 429 SCSI driver 440<br />
<br />
shared memory segment 450 shmSize 450 soft memory limit 451 swap partition 435 swapping 428, 438 terminal servers 441 timeout, authentication 451 trunking 429, 437 tuning 427 virtual terminals, disable 449 VirtualCenter 580 VM memory allocation 439 VM tuning 439 VMFS 429, 435 vmkusage 588 ESX Server 2.5 design 452 disk drives 445 disk partition sizes 445 farm 456 Fibre Channel adapters 443 IP Out 459 license 453 load balancing 458 MAC Out 458 maximum queue depth 437 network adapter usage 443 networking 457 NIC configuration 446 outstanding disk requests 438 partition sizes 445 PCI adapter placement 443 rule of thumb 453 SchedNumReqOutstanding 438 service console memory 446 service console networking 457 sizing 453 specifications 452 storage configuration 437 storage sizing 457 time sync 442 Virtual Infrastructure 453 virtual machine networking 458 VMotion 456, 458 vSwitch usage 444 ESX Server 3 4-Way virtual SMP 460 4-way Virtual SMP 459 Consolidated Backup 460<br />
<br />
Index<br />
<br />
1045<br />
<br />
disk partitioning 465 Distributed Resource Scheduler 460 Fibre Channel adapters 463 firewall 467 High Availability 460 hot-add disks 461 iSCSI 467 iSCSI support 459 NAS support 459 new features 459 partitioning 465 PCI adapter placement 463 service console changes 461 service console firewall 467 service console memory 464 swap partition 465 virtual machine memory 460 virtual switch 466 VMFS 3 461 VMFS partitions 464 VMware HA 460 vSwitch policy exceptions 467 what’s new 459 esxcfg-firewall command 468 esxtop 580 batch mode 585 columns to display 585 commands 584 CPU information 584, 586 disk information 584 example 434 exit 587 logging 585 memory information 584, 587 network information 584 PCPU usage 586 starting 581 Ethernet 235–292 See also network subsystem 1480 byte packet size 245 jumbo frames 255 linear scaling 249 examples 707 Exchange Server 831–845 /3GB switch 835 Active Directory 831 Analyzer 833 backup 844 Best Practices Analyzer Tool 833<br />
<br />
1046<br />
<br />
boot.ini file 835 counters 838 CPU subsystem 840 data placement 842 defragmentation 843 design 833 disk subsystem 841 ESE buffer 838 Exchange 2007 845 expired messages 843 Global Catalog 834 HeapDecomitFreeBlockThreshold 837 Information Store 843 IOAT 835 IPSec 835 mailboxes 843 memory subsystem 835 migration 833 msExchESEParamCacheSizeMax 839 network infrastructure 833 network subsystem 834 paging file 835 planning 832 processor subsystem 840 RAID levels 841 TOE 835 userva switch 837 versions 832 Windows support 832 EXEC_PAGESIZE 694 Execute Disable Bit 153 EXP300 enclosure 199 EXP400 enclosure 199 Explicitly Parallel instruction Computing 60 Ext3 file system 397<br />
<br />
F FB-DIMMs 137 fiber optic cabling 187 Fibre Channel See also disk subsystem See also DS4000 balancing the I/O load 230 cache hit percentage 232 components 187 DS4000 See DS4000 I/O operation 220<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
I/O request rate 231 I/O size 221, 223 iSCSI comparison 189 performance factors 188 protocol layers 222 RAID levels 232 redundant paths 187 rules of thumb 226 SATA comparison 226 scalability 187 SCSI protocol 223 SCSI, comparison with 186 segment size 208, 233 Storage Manager 229–234 throughput 221, 225 transfer rate 231 file server 15, 745 foreground boost 751 important subsystems 746 Linux 759 logging 760 performance, effect on 747 stripe size 209 system cache 751 virtual memory 752 Windows 750 file system cache, Windows 309 firewall 467 floating-point operations 37 Foster 42 CPU subsystem Foster MP 44 four phases 4 fragmentation, disk 361, 363 frames, Ethernet 237 free command 554, 695 fsutil command 359 Fully Buffered DIMMs 33, 137<br />
<br />
G Gallatin 44 CPU subsystem 44 Gigabit Ethernet See also network subsystem 10 Gigabit Ethernet 256 checksum offload 259 packets per second 676 Greencreek 114<br />
<br />
groupware servers 20 guidelines 5<br />
<br />
H hardware assists 82 hardware compatibility test IBM Center for Microsoft Technologies 7 Hardware Prefetcher 74 hardware scalability 100 High Availability 460 High Performance Computing 640 high performance computing 26 home node 430 HPMA 161 HugeTLBfs 394 Hyper-Threading 39–41, 656 DB2 850 ESX Server 430 interrupt processing 333 kernel selection 389 Linux 388 network performance 253 optimized 40 software scalability 109 Xeon DP 42 HyperTransport 123–124, 126 hypervisor 81<br />
<br />
I I/O Acceleration Technology 267 See also IOAT IA-32e mode 63 IA64 60 IBM Center for Microsoft Technologies 7 IBM DB2 See DB2 IBM Director See also Capacity Manager ESX Server 449 IBM TotalStorage DS4000 See DS4000 IBM Xcelerated Memory Technology 164 IIS See Microsoft IIS InfiniBand 27, 93, 290 initial placement, ESX Server 431 Intel 5000 chipset 114<br />
<br />
Index<br />
<br />
1047<br />
<br />
Advanced Digital Media Boost 52 Advanced Smart Cache 51 chipsets 98 Core microarchitecture 50 Data Execution Prevention 153 E7520 112 E8500 chipset 121 EM64T 62 Execute Disable Bit 153 Hyper-Threading See Hyper-Threading Intel 64 Technology 62 Intelligent Power Capability 51 Itanium 2 59 Smart Memory Access 52 Virtualization Technology 82 VTune 535 Wide Dynamic Execution 50 Xeon DP processor 42 Xeon MP 43 Intel 64 Technology addressable memory 155 architecture 62 Linux 389 modes 62 interleaving interleave depth (disk) 207 memory 143 Internet Explorer Performance console, use with 503 interrupt assignment 332 INTFILTR utility 332, 680 introduction 3 IOAT 267 adapters supported 268 Clovertown processors 49 data flow 268 implementation 267 operating system support 271 SLES 10 415 TOE comparison 271 iostat command 543, 700 IP datagrams 237 IPSEC, iSCSI 190 IPX 684 Irwindale 43 isag command 693 iSCSI 171, 189, 283 encapsulation 286<br />
<br />
1048<br />
<br />
encryption 190 ESX Server 467 Fibre Channel comparison 189 hardware initiator 287 host bus adapter 288 infrastructure 288 initiator 284 latency 190, 288 NAS comparison 284 network 288 network load 288 OSI model 285 performance 189 remote boot 288 SCSI comparison 189 security 288 session layer 285 software initiator 286 TCP/IP packets 286 technology 283 throughput 190 TOE, combined with 287 Itanium 2 59–61 32-bit applications 61<br />
<br />
J journaling Linux 395 options for Ext3 401 jumbo frames 255<br />
<br />
K KDE System Guard 547–553 memory monitoring 695 network bottlenecks 703 kernel /proc file system 382 Hyper-Threading 389 parameters 381, 384 powertweak 381 selection 389 sysctl command 383 kernel swap behavior 393 Kirkland Programming Center 7 Knowledge Consistency Checker 14 kswapd 393<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
L lanes 90 LargeSystemCache 313 last access time Domino 792 Linux 398 Windows 358, 755 latency disk 170 memory, memory 144 NUMA 102 latent bottlenecks 635 legacy mode 62 Lindenhurst 112 Linux 371–424 /proc 382 accept_redirects 408 access time updates 398 affinity 693 analyzing bottlenecks 687–705 Capacity Manager 690 chkconfig command 374 CPU affinity 693 CPU bottlenecks 692 CPU subsystem 388 daemons, disabling 372 data placement 750 disk bottlenecks 698 disk subsystem 395 dmesg command 540 Domino 773 elevator algorithm 399 elvtune command 399 EM64T support 389 EXEC_PAGESIZE 694 Ext3 file system 397 file server 745, 759 file system 395 free command 554, 695 GUI, do not run 376 Hyper-Threading 41, 388 icmp_echo_ignore_broadcasts 408 init command 377 IOAT support 271 iostat command 543, 700 ipfrag_low_thresh 411 isag command 693 journaling 395 KDE System Guard 547–553<br />
<br />
memory monitoring 695 network bottlenecks 703 kernel parameters 384 which one to use 389 last access time 398 level2 oplocks 759 logging 760 Lotus Domino 773 memory botlenecks 694 memory bottlenecks 694 memory subsystem 395 mingetty 378 mpstat command 560 network bottlenecks 703 network subsystem 407 nice command 542 nmon 701 noatime 398 notail 402 oplocks 759 page size 694 partitioning recommendations 403 performance bottlenecks 687–705 PLPerf 561 pmap command 557 RAID levels 750 ReiserFS file system 398 rmem_max 410 rp_filter 410 Samba 759 sar command 546, 693 secure_redirects 407 Security Enhanced Linux 379 send_redirects 408 sleep mode 696 SMP-based systems 693 socket options 760 Static Resource Allocation Table 103 strace command 558 stripe size 209, 402 swap partition 405 sysctl command 383 Sysstat package 537 tagged command queueing 402 taskset command 693 tcp_fin_timeout 410 tcp_keepalive_time 410 tcp_max_syn_backlog 411<br />
<br />
Index<br />
<br />
1049<br />
<br />
tcp_rmem 410 tcp_tw_recycle 408 tcp_tw_reuse 408 tcp_wmem 410 TIME-WAIT 408 TOE support 266 tools 537–561 top command 388, 541, 693 Traffic-vis utility 554, 704 ulimit command 559 uptime command 539, 692 vmstat command 545, 693, 699 wmem_max 410 write back cache 750 xPL 561 zombie processes 543 Linux Technology Center 7 load balancing 431, 458 local memory 149 logical drive configurations 206 logical drive migration 203 logical processors 39 long mode 63 Longhorn 368 Lotus Domino 763–830 $Revisions field 794 $UpdatedBy field 793 Agent Manager 802, 804 concurrent agents 806 when it runs 805 agents 820 alarms 802 availability threshold 826 balancing workload 826 cache 770 caching of databases 777 calconn 802 calendaring and scheduling 801 clustering 826 compact command 778 compression 819, 822 CPU subsystem 764 database performance 777 Database.DbCache.Hits 777 directory indexer 781 disk subsystem 765 display images after loading 790 document table bitmap optimization 792 file-max parameter 773<br />
<br />
1050<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
fixup command 778 freetime 801 full-text index 781 future growth 828 headline monitoring 793 IIOP 813 images, displaying 789 IMAP 813 indexer 780 InitialDbOpen 777 iNotes 821 Internet Site documents 813 kernel 768 large address aware 770 LargeSystemCache 772 LastAccessed property 792 LDAP Site documents 813 Linux 773 load balancing 826 logging 767 mail performance 795 mail poll time 809 mail threads, maximum 795 mail.box, multiple 799 MailMaxConcurrentXferThreads 797 MailMaxDeliveryThreads 796 MailMaxThreads 795 maintain LastAccessed property 792 maintenance 827 maximize data throughput for... 771 maximum size of request content 818 memory 765 memory allocation 768 memory management 787 memory mapped I/O 166 memory mapping 766 message caching 798 MinNewMailPoll 809 network 764 network compression 822 network subsystem 768 network timeouts 814 No_Force_Activity_Logging 803 NoMsgCache 798 NOTES.INI 774 NRPC 823 NSF_Buffer_Pool_Size 777 NSF_DbCache_Disable 778 NSF_DbCache_Maxentries 777<br />
<br />
NTFS file allocation size 773 paging file 766 partitioning 825 performance 774 per-user message caching 798 planning 828 POP3 813 port encryption 825 private memory 769 RAID, use of 766 registry settings 772 replication performance 810 replication tasks 811 Replicator task 776 resource reservations 802 response hierarchy 793 room reservations 802 Router task 776 Runtime/Restart performance 812 Sched 802 Server Configuration document 775 server tasks 775 Server_Availability_Threshold 827 Server_MaxSessions 807–808 Server_MaxUsers 809 Server_Session_Timeout 808 session timeout 808 sessions, concurrent 807 SET CONFIGURATION command 775 shared memory 769 shmmni parameter 774 SHOW CONFIGURATION command 774 SMTP 813 specialized response hierarchy 793 Statistics & Events database 811 Stats task 776 stripe size 209, 767 subsystems, important 764 system cache 770 tasks compact 792 IMAP 798 Replicator 776 Router 776 Stats 776 threads 817 threads-max parameter 774 timeouts 814 traffic considerations 827<br />
<br />
transactional logging 811–812 type-ahead addressing 798 unread marks 790 updall command 778 Update task 779 users, maximum 809 virtual memory 769 Web access 821 Web agents 820 Web server performance 813 Web Site documents 813 workload balancing 826 Low Voltage Differential 171<br />
<br />
M mail server 18 maximize throughput for file sharing 310 memory mapped I/O 166 memory subsystem 129–167 4 GB, more than 151 add memory 664 addressability 66 addressable memory 155 advanced ECC memory 156 Advanced Memory Buffer 137, 139 analysis example 731, 741 AWE 154 bandwidth 144 BIOS settings 161 BIOS, impact of 165 bottlenecks 645, 879 buffered 134 cache line 144 capacity 131 CAS 145 case study 741 Chipkill 156 clock cycles 144 DDR memory 135 DDR2 memory 135 DDR2 performance 136 DIMM location 142, 165 DIMMs 131 DRAM chips 131 ECC 131, 156 FB-DIMM performance 140 FB-DIMMs 137 HPMA 161<br />
<br />
Index<br />
<br />
1051<br />
<br />
interleaving 143 introduction 32 latency 144, 146 Linux 395 Linux bottlenecks 694 loaded latency 146 local memory 149 location of DIMMs 142 maximum memory addressable 66 memory mapped I/O 166 memory mapping 766 Memory ProteXion 163 mirroring 33, 157 NUMA 149 Opteron 150 PAE 152 paged and non-paged RAM 662 paging to disk 166, 666 PC1600-PC3200 specifications 141 peak throughput 141 performance analysis 645 performance gains 167 primary counters 638 processor performance, affect on 110 rank 133 RAS 145 registered 134 remote memory 149 rules of thumb 165–166 SDRAM 134 buffered/unbuffered 134 STREAM benchmark 147 technology 131 timing 142 tuning options 664 types 130 types of memory 130 unbuffered 134 upgrading, effect of 749 utilization rules of thumb 166 virtual memory 663 Windows bottlenecks 661 working set 165 worksheet 645 x3950 160 x3950 rules 160 Xcelerated Memory Technology 164 MESI 105 Microsoft IIS 981–1016<br />
<br />
1052<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
ActivityPeriod 988 apartment-threaded 991 AppAllowDebugging 989 application tuning 1009 applications pool 997 ASP buffering 1010 ASP caching 1011 ASP debugging 1012 AspBufferingOn 990 AspDiskTemplateCacheDirectory 990 AspExecuteinMTA 991 AspMaxDiskTemplateCacheFiles 991 AspProcessorThreadMax 992 AspQueueConnectionTestTime 992 AspRequestQueueMax 993 AspScriptEngineCacheMax 993 AspScriptFileCacheSize 994 AspSessionMax 994 AspSessionTimeout 995 AspTrackThreadingModel 995 cache management 985 cache purging 1007 cache size 1007 CacheISAPI 996 CentralBinaryLoggingEnabled 996 compression 1002 connection management 986 cookies 1009 counters 1013 CPU monitoring 1000 CPU subsystem 1004 DataSetCacheSize 989 DisableLazyContentPropagation 996 DisableMemoryCache 987 DisableMemoryCache setting 1007 disk subsystem 1007 free-threaded 991 health options 1001 HTTP keep-alive 1004 http.sys 982 introduction 982 isolation 983 keep-alive 1004 kernel mode tuning 984 kernel-mode response cache 982 logging 1008 MaxCachedFileSize 987, 1007 MaxConnections 986 maximize data throughput for 1006<br />
<br />
MaxPoolThreads 987 MemCacheSize 987, 1007 memory subsystem 1005 metabase 989 Metabase Explorer 989 network subsystem 1002 NLB 1020 ObjectCacheTTL 988, 1007 performance counters 1013 performance options 999 pinging 1001 PoolThreadLimit 988 processor subsystem 1004 RAID 1007 rapid-fail protection 1001 recycling options 997 registry keys 985, 987 request management 986 request queue limit 999 response cache 982 session state 1009 shutdown time limit 1001 shutdown worker processes 999 SSL 1005 startup time limit 1001 static content 1002 tuning applications pool 997 kernel mode 984 user mode 986 UriEnableCache 985 UriMaxCacheMegabyteCount 985 UriMaxCacheUriCount 986 UriMaxUriBytes 985 UriScavengerPeriod 985 user mode tuning 986 Web Capacity Analysis Tool 982 Web garden 1000 worker process isolation mode 983 worker processes 982 Microsoft Management Console 474 Microsoft Scalable Networking Pack 273 Microsoft SQL Server See SQL Server Microsoft Word Performance console, use with 503 mirroring, memory 157 MMC 474 MOESI 106<br />
<br />
Molex cable 177 monitoring tools ESX Server 579–589 Linux 537–561 Windows 471–535 mpstat command 560 MTU size Linux 384 Windows 347 multimedia servers 21 Myrinet 27, 289 mySAP ERP memory mapped I/O 166<br />
<br />
N NAS 185 iSCSI comparison 284 NET SERVER CONFIG command 364 NetBEUI 684 NetBIOS 684 NetBurst architecture 36 NetWare stripe size 209 network adapter usage 443 Network Analyzer, use of 649 Network Load Balancing 12 Windows Terminal Services 977 Network Monitor 511 capturing network traffic 515 configuring filters 514 filters 514 installing 512 packet analysis 518 promiscuous mode 511 raw data 519 starting 513 System Management Server 512 tips 519 using 513 versions 512 viewing data 517 network monitor driver 502 network subsystem 235–292 See also iSCSI 10 Gigabit Ethernet 256 1480 byte packet size 245 adapter command overhead 237 adapters 236<br />
<br />
Index<br />
<br />
1053<br />
<br />
auto-negotiation 325 bcopies 251 binding order 682 bottlenecks assumptions 237 finding 649 solving 678 two types 674 busmaster adapter 237 cabling 683 checksum offload 259 Chimney Offload 273 command overhead 237 CPU count 252 CPU performance 251 design 682 DMA transfers 237 network subsystem 240 duplex setting 325 Ethernet frames 237 frame maximum 245 size 242 Hyper-Threading 253 InfiniBand 290 interrupts handled by a specific CPU 332 Intfilter utility 332 INTFILTR 332, 680 introduction 33 IOAT 267 IOAT-TOE comparison 271 iSCSI 283 jumbo frames 255 large packet sizes 245 limiting factors 237 linear scaling 249 link speed setting 325 Linux 407, 703 memory copies 240, 251 multiple ports 246 Myrinet 289 NetBEUI 684 network monitor driver 502 number of processors 252 packet defined 237 packets per second 649, 676 packets per second limit 237 size 242, 258<br />
<br />
1054<br />
<br />
PCI bus 240 PCI busmaster 237 performance 236, 242, 649 ports, multiple 246 primary counters 638 processor speed 251 protocols 682–684 receive-side scaling 249, 276 Remote DMA 280 Scalable Networking Pack 273 small packet sizes 244 SMP scaling 258 summary 257 TCP Chimney Offload 273 TCP offload engine 260 TCP/IP performance 238 Windows 685 TOE 260 TOE-IOAT comparison 271 transfer size 242, 258 tuning options Linux 704 Windows 674, 678 upgrading, effect of 748 Windows bottlenecks 672 Windows protocols to remove 318 worksheet 649 xcopy, use of 257 network-attached storage 185 NICconfiguration 446 nice command 542 nmon 571–577, 701 Analyser Excel macro 576 batch mode 575 command-line 575 count 575 data collection mode 575 download 572 Excel macro 576 file name 575 graphs 576 interactive mode 572 interval 575 Linux 571 nmon2csv 575 using 572, 701 Nocona 43 Notes<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
See Lotus Domino NOTES.INI 774 NTFS cluster size 757 compression 360, 758 Last Access Time 358 log size 758 tuning 756 use of 359 NUMA 149 DB2 849 defined 101 Linux 390 Linux kernel 389 Opteron 125 VMware ESX Server 430 Windows 657 NWLink 684<br />
<br />
O OLTP 929 on-going performance analysis 4 operating system levels 80 operating systems ESX Server 425–468 introduction 34 Linux 371–424 Windows Server 2003 295–370 Opteron 52 AMD64 53 block diagram 126 DDR2 support 54 HyperTransport 56, 124, 126 memory access 124 memory addressable 66, 155 memory subsystem 150 NUMA 125 NX feature 153 Pacifica 54 Passthru card for x3755 56 PCI Express support 54 Revision E 53 Revision F 54 specifications 53 SUMO 103, 125 Oracle 923–949 application tuning 923 architecture 924<br />
<br />
archiver process 927 background processes 927 basic components 924 block size 943 buffer cache 934 architecture 934 hit-ratio 934 buffer pools, multiple 936 checkpoint process 928 control files 928 counters 947 data files 928 database block size 943 buffer cache 926, 933 structure 925 writer process 927, 933 DB_BLOCK_BUFFERS 934 DB_BLOCK_SIZE 934 dedicated server processes 927 default pool 936 dictionary cache 926, 933 dictionary segments 944 disk controller cache size 939 disk subsystem 937 disk tuning 924 Dynamic Performance Views 947 execution of statements 926 INIT.ORA 928 keep pool 936 library cache 926, 933 log writer process 927, 937 memory 925 memory tuning 923, 932 monitoring performance 946 OLAP 929 OLTP 929 operating system tuning 923 Oracle Performance Monitor 947 Oracle Tuning Pack 946 Performance Monitor 947 process monitor process 928 processes 924, 927 processors, number of 73 protocol binding order 932 protocols 932 RAID levels 940, 945 read-only segments 944 recovery process 928<br />
<br />
Index<br />
<br />
1055<br />
<br />
recycle pool 936 redo log buffer cache 925, 937 redo logs 928 resource contention tuning 924 rollback segments 944 segments 944 server type 17 services, stopping 932 SGA 925 shared pool 926, 933, 948 SHARED_POOL_SIZE 948 sort area size 948 Statspack 947 stripe size 209, 942 subsystems, important 931 System Monitor process 928 system rollback segments 944 table spaces 943 tables 926 temporary segments 944 tuning steps 923 virtual performance tables 947 Windows 2000 settings 931 OSI model 237<br />
<br />
P Pacifica 83 packet defined 237 segmentation 258 PAE defined 152 parameter in BOOT.INI 336 Windows 335 PAGEFILE.SYS 305 paging case study 739 disable kernel paging 351, 355, 976 Domino 766 Linux 694 RAID-5 not recommended 307 unavoidable 666 Windows 305 Parallel ATA 178 paravirtualization 82 Passthru card for x3755 56 Paxville 46 PC1600-PC300 memory specifications 141<br />
<br />
1056<br />
<br />
PCI adapter placement 443 PCI Express 32, 90 AMD Opteron support 54 bandwidth 92 bridge not required 95 compared with PCI-X 92 lanes 90 link 90 overhead 93 performance 93 physical size 92 slot compatibility 92 uses 93 x3650 block diagram 96 PCI subsystem 85–96 See also PCI Express See also PCI-X agent 86 bridges 94 busmaster devices 85 design 86 initiators 86 introduction 32 modes 89 multiple buses 94 multiplexed address and data bus 86 notches in PCI adapters 89 PCI transaction 86 performance 90 targets 86 turnaround phase 86 PCI-X 86 See also PCI adapter keying 89 attribute phase 87 bridging 95 compared with PCI 87 disconnect boundary 88 frequencies 88 modes and speeds 89 performance 90 split transactions 87 throughput 87 x3650 block diagram 96 perfmon See Performance console performance analyzing 633 bottlenecks, finding 627<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
cache size 71 Capacity Manager 591 case studies 707 chipsets 100 Citrix MetaFrame 953 CPU clock speed 72 CPU subsystem 66, 100 data set size 203 DDR2 136 disk subsystem 200, 205 Domino 828 drives, number of 202 Ethernet adapter 236 FB-DIMMs 140 Fibre Channel 220, 224 file server 747 Hyper-Threading 40 I/O transfer size 221 interleaving 143 IOAT 268, 270 latent bottlenecks 635 logical drive configurations 206 Lotus Domino 774 memory bandwidth 144 network subsystem 235 page file 307 PCI Express 93 primary counters 638 RAID adapter cache size 216 RAID rebuild time 217 RAID strategy 201 SCSI bus speed 213 SCSI buses, multiple 214 spread of data 203 stripe size 207 TCP/IP 238 TOE 260, 263 tools ESX Server 579–589 Linux 537–561 Windows 471–535 transfer size for Ethernet 242 well balanced system 635 Windows Terminal Services 953 write-back versus write-through 214 X3 Architecture 73 Performance console 472–505 adding counters 484 alerts 493<br />
<br />
actions 479, 495 creating 493 deleting 497 importing 497 saving 496 schedule 495 starting 496 threshold values 494 BLG file format 489 Capacity Manager, compare with 472 chart view 476, 481 counter log 477, 486 creating 487 deleting 491 file formats 489 importing 491 saving 491 starting 490 time frame of view 492 counters 480 Active Server Pages Request Execution Time 1016 Request Wait Time 1016 Requested Queued 1016 Requests Executing 1015 Requests/sec 1015 Transaction/sec 1016 Logical Disk % Disk Time 918 Memory Available Bytes 638, 731 Available MBytes 646, 663–664 Cache Faults/sec 711 Page Faults/sec 918 Page Reads/sec 638, 645, 663–664, 710, 731, 741 Page Writes/sec 638, 646, 663–664, 731 Pages/sec 879, 918 Pool Nonpaged Bytes 646, 664 NetBEUI Bytes Total/sec 684 Datagrams/sec 684 Frames/sec 684 Network Interface Bytes Received/sec 650, 673, 709 Bytes Sent/sec 651, 673 Bytes Total/sec 638, 650, 673 Output Queue Length 685<br />
<br />
Index<br />
<br />
1057<br />
<br />
Packets Received/sec 651, 673 Packets Sent/sec 651, 673 Packets/sec 639, 651, 673 Oracle Buffer Cache %physreads/gets 948 Oracle Data Dictionary Cache %getmisses/gets 948 Oracle Data Files phyreads/sec 948 Oracle Dynamic Space Management %recursive calls/sec 948 Oracle Library Cache %reloads/pins 948 Oracle Redo Log Buffer redo log space requests 948 Oracle Sorts sorts in memory/sec 948 sorts on disk/sec 948 Page Reads/sec 709 Page Writes/sec 709 Paging File % Usage Peak 646 %Usage Max 308 PhysicalDisk Avg. Disk Bytes/Read 733 Avg. Disk Bytes/Transfer 210, 225, 644, 669, 739 Avg. Disk Bytes/Write 733 Avg. Disk Queue Length 644, 669, 738, 902–903, 917 Avg. Disk sec/Read 715 Avg. Disk sec/Transfer 638, 644, 652, 669 Avg. Disk sec/Write 709, 716, 734 Current Disk Queue Length 902–903 Disk Bytes/sec 225, 644 Disk Transfers/sec 734 Split IO/sec 644 Process Page Faults/sec 918 Virtual Bytes 838 Processor % Privileged Time 647, 658 % Processor Time 638, 647, 658, 709, 712, 730, 737, 918 % User Time 647, 658 Interrupts/sec 648 Server Pool Nonpaged Failures 646<br />
<br />
1058<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Pool Nonpaged Peek 646 SQL Server 916 Access Methods - Page Splits 914 Buffer Cache Hit Ratio 917 Buffer Manager<br />
<br />
Buffer Cache Hit Ratio Database pages 879 Free Buffers 901 Read-Ahead Pages 905<br />
<br />
876<br />
<br />
BufferNode 879 Cache Hit Ratio 918 Free Pages 720 I/O Single Page Writes/sec 919 I/O-Pages Reads/sec 919 I/O-Transactions/sec 918 Page Reads/sec 917 Page Writes/sec 917 SQL Compilations/sec 917 Total Server Memory 917 User Connections 917, 919 System % Total Processor Time 918 Processor Queue Length 648, 659, 918 TCP Segments Retransmitted/sec 685 Segments/sec 685 Terminal services Active Sessions 980 Terminal services session % Processor Time 980 Total Bytes (per second) 980 Working Set 980 UDP Datagrams/sec 685 Web Service Bytes Total/sec 1015 Current Connections 1015 File Cache Hits% 1015 Kernel URI Cache Flushes 1015 Kernel URI Cache Hits% 1015 Kernel URI Cache Misses 1015 Total Method Request/sec 1015 CPU bottlenecks 657 CSV file format 489 database servers, use with 503 deleting objects 485 disabled counters 505 disk counters 502<br />
<br />
DISKPERF command 212, 502, 917 ETL file format 500 event providers 479 Explain button 484 Fibre Channel 225 functions 477 highlighting a counter 485 histogram view 476, 483 icons 482, 487 instances 480 Internet Explorer, use with 503 Linux 561 logical drive counters 502 LogicalDisk object 667 logs 486 Microsoft Word, use with 503 network counters 502 objects 479 overhead of using 473 overview 475 Performance Logs and Alerts 477, 486 physical drive counters 502 PhysicalDisk object 667 providers 479 remote machines, accessing 486 report view 476, 483 resource kit tools 520 schedule 490 settings, saving 485 spreadsheet applications, use with 503 starting 474 System Monitor 476, 481 toolbar 482, 487, 493, 497 trace log advanced settings 501 buffer settings 501 creating 498 events 499 file formats 500 provider 499 schedule 500 trace logs 477, 487, 497 TSV file format 485, 489 views 476 word processors, use with 503 Performance Lab 5 Performance Monitor Linux 561 performance tuning 629<br />
<br />
phases 4 ping command 347 pipeline, CPU 37 PLPerf 561 See also xPL pmap command 557 Potomac 44 preemptive multitasking 303 prefetch 74 Prestonia 42 primary counters 638 print servers 16, 761 privilege levels 80 processor subsystem See CPU subsystem protocol layers, Fibre Channel 222 protocols, network 683<br />
<br />
Q quad-core processors 49 questions to ask 631<br />
<br />
R Radware 12 RAID array 4 RAID levels 191 composite RAID levels 199 Fibre Channel 232 page file recommendation 307 RAID-0 192 RAID-00 200 RAID-1 192 RAID-10 199 RAID-1E 193 RAID-1E0 200 RAID-4 194 RAID-5 194 not for page files 307 RAID-50 200 RAID-5E 195 RAID-5EE 195 RAID-6 198 rebuild time 217 strategy 201 RAID-M 157 rank 133 RAS 145 rebalancing 432<br />
<br />
Index<br />
<br />
1059<br />
<br />
rebuild time 217 receive-side scaling 249, 276 Red Hat Enterprise Linux See also Linux daemons 372 Ext3 file system 397 hugetlb_pool 387 inactive_clean_percent 387 inet_peer_gc_maxtime 387 pagecache 387 Redbooks Web site 1030 Contact us xxix redundant bit steering 163 redundant paths 187 registered SDRAMs 134 registry parameters 754 ReiserFS 398 notail 402 Remote Differential Compression 14 Remote DMA 280 remote memory 149 RETAIN tip 46 rings 80 rotational positioning optimization 206, 642 RTP Performance Lab 5 rules of thumb cache size 71 CPU performance 76 disk subsystem 228, 643 Fibre Channel 226 memory for x3950 160 memory subsystem 165–166 write-back versus write-through 216<br />
<br />
S Samba 745, 759 SAN 183 backend zone 183 design 183 frontend zone 184 sar command 546, 693 SAS 174 See also disk subsystem cables 176 components 174 defined 170 introduction 33 lanes 176<br />
<br />
1060<br />
<br />
layers 175 number of drives 178 protocols 175 SCSI comparison 177 speed negotiation 176 SATA 176, 178–182 defined 170 Fibre Channel comparison 226 introduction 33 scalability 12 scalability directory 118 Scalable Networking Pack 273 scale-up 100 versus scale-out 12 scenarios 707 screen savers 363, 932 SCSI 170 array controller operation 172 Fibre Channel, comparison 186 iSCSI comparison 189 Logical Block Address 172 performance 213 rotational latency 173 SAS comparison 177 speed, affect on performance 213 SDRAM 134 See also memory subsystem SEC 156 sector prefetch 74 seek operation 172 seek time 170 segment size, Fibre Channel 208 segments, TCP 237 SELinux 379 Serial ATA 178–182 features 179 in the enterprise 181 ServeRAID 180 standard 178 server selection 4 server types 11 Active Directory server 14 communication servers 21 database servers 17 DHCP servers 24 DNS servers 24 domain controllers 13 e-mail servers 18 file server 15, 745–761<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
groupware servers 20 HPC 26 multimedia servers 21 print servers 16, 761 terminal server 22 virtualization servers 26 Web servers 19 WINS servers 25 worksheet 634 ServeRAID See also RAID See also SAS See also SCSI See also Serial ATA cache size 216 firmware 220 large arrays 199 logical drive migration 203 operation 172 rebuild time 217 Serial ATA adapter 180 stripe size 207, 209 write-back versus write-through 214 ServerWorks chipsets 98 ServerWorks Grand Champion 4.0 HE/LE 111 service console, ESX Server firewall 467 memory 446, 464 set associativity 67 shared L3 cache 47 SIMD 38 single core 36 single-rank DIMMs 133 sizing 453 Sledgehammer 52 SLT files 593 SMP 101, 147 effect on performance 72 snoop cycles 105 snoop filter 118 software considerations 657 software scalability 108 SQL Server 861 64-bit versus 32-bit 870 affinity 887 network 892 processor 889 ALTER DATABASE command 883<br />
<br />
analysis 709 Analysis Services 867 async I/O 862 AWE 872 AWE support 906 Books Online 865 boost SQL Server priority 887 buffer cache 875 Buffer Cache Hit Ratio 876 cache usage 885 checkpoint 862, 902 clustered indexes 905, 909, 911 concurrent users 919 configuration disk 881 memory 876 context-switching 888 counters 879, 916 See Performance monitor, counters covering indexes 910 CPU 884 Data Transformation Services 868 database 874 Database Engine 865 Database Engine Tuning Advisor 919 disk 875 disk configuration 881 disk partitioning 881 disk subsystem 881 DROP_EXISTING 914 DTA 919 dynamic management views 920 dynamic memory management 877 editions of 2005 865 Enterprise Edition 863, 865 example 709 features 864 federated servers 863 fibers 888 file system cache 897 FILLFACTOR 914 fixed memory size 879 hashed page 905 hot-add memory 866 IAM 905 index allocation map 905 Index Tuning Wizard 875 indexes 875, 908, 911 Integration Services 868<br />
<br />
Index<br />
<br />
1061<br />
<br />
LazyWriter 862, 901 light-weight pooling 888 log files 882 log manager 862, 904 max async I/O 900 maximum user queries 887 MDAC 868 memory 875–876 memory configuration 878 memory limit 863 minimum query memory 879 mirroring 866 network affinity 892 network subsystem 891 non-clustered indexes 908 Notification Services 868 online indexing 866 online restore 866 PAD_INDEX 914 PAE 871 paging 875 parallelism 886, 889 partitioning 866, 881 performance 874 Performance console objects 875 processor 884 processor affinity 889 processor configuration 886 processor scheduling 896 processors, number of 73 Profiler 875 Query Analyzer 875 query hints 921 questions 874 RAID levels 882 read-ahead manager 862, 905 REINDEX command 914 replication 868 reserve physical memory 879 SAN 881 scaling 895 security 867 separate data 882 server type 17 Service Broker 869 SHOWCONTIG command 914 snapshot isolation level 867 SQL Client 864 SQL Profiler 919<br />
<br />
1062<br />
<br />
SQL Trace 919 SQLCTRS.PMC 917 SQLdiag utility 921 SQLOS 864, 869 SQLSERVR process 918 Standard Edition 864 stripe size 209 subsystems, important 876 system cache 897 tempdb database 883 thread mode 888 tuning 875 tuning Windows 896 Tuning Wizard 919 user connections 875 version selection 870 versions 863 virtual memory 898 worker threads 887 x64 873 SSA 170 Static Resource Allocation Table 103 static Web pages 19 Storage Attached Network 182 strace command 558 STREAM benchmark 147 stripe size 207 affect on performance 642 case study 739 Fibre Channel 233 Linux 402 Lotus Domino 767 page file 212 video file server 209 Web server 209 stroke 203 subsystems, important Active Directory server 14 communication servers 21 database servers 17 DB2 847 DHCP servers 24 DNS servers 24 domain controller 13 file servers 15 groupware servers 20 HPC servers 26 mail servers 18 multimedia servers 21<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Oracle 931 print server 16 SQL Server 876 terminal server 22 virtualization servers 26 Web servers 19 WINS servers 25 SUMO 103, 125 SUSE Linux Enterprise Server See also Linux 10 411 accept_redirects 385 autoconf parameter 385 dad_transmits 385 daemons 373 heap-stack-gap 386 IOAT 415 ip_conntrack_max 384 powertweak 381 regen_max_retry 385 ReiserFS file system 398 router_solicitation_delay 385 router_solicitation_interval 385 router_solicitations 385 sched_yield_scale 384 shm-bigpages-per-file 384 shm-use-bigpages 384 temp_prefered_lft 385 temp_valid_lft 385 Traffic-vis utility 554 virtualization 420 vm_anon_lru 386 vm_lru_balance_ratio 386 vm_mapped_ratio 386 vm_passes 386 vm_shmem_swap 386 vm_vfs_scan_ratio 386 Xen 411, 420 YaST 375 SYN requests 342 sysctl commands 383 Sysstat package 537 System Monitor See Performance console System x Performance Lab 5 System x Performance Logger 561 See also xPL System x3755 Passthru card 56<br />
<br />
T Task Manager 505–511 columns 507 performance 509 processes 506 starting 505 taskset command 693 TCP Chimney Offload 273 TCP offload engine 260 See also TOE TCP segments 237 TCP table 344 TCP/IP 238, 685 counters 685 Linux kernel parameters 384 MaxUserPort 343 MTU size Linux 384 Windows 347 operations 238 Path MTU 349 TCP acknowledgement frequency 346 TCP connection retransmissions 342 TCP Control Block table 344 TCP data retransmissions 342 TCP window scaling 340 TCP windows size 339 TIME-WAIT Linux 408 Windows 343 Windows tuning 338 Tigerton 50 time sync 442 TIME-WAIT Linux 408 Windows 343 TOE 260 adapter support 266 benefits 261 data flow 261 IOAT comparison 271 iSCSI, combined with 287 operating system support 266 purpose 260 throughput 263 tools ESX Server 579–589 Linux 537–561 Windows 471–535<br />
<br />
Index<br />
<br />
1063<br />
<br />
top command 388, 541, 693 TotalStorage DS4000 See DS4000 TRACERPT 477 Traffic-vis utility 554, 704 Translation Lookaside Buffer 394 Tulsa 47 Tumwater 112 Twin Castle 121<br />
<br />
U ulimit command 559 unbuffered/unregistered SDRAMs 134 uptime command 539, 692 user authentication 14<br />
<br />
V VBScript 522 Video file server 209 video subsystem 34 Virtual Infrastructure Node 453 virtual memory, Windows 305 virtual switch 466 VirtualCenter 453 new features 459 what’s new 459 virtualization 80 Xen 420 virtualization hardware assists 79–84 virtualization servers 26 virus scanner applications 363 VMFS 3 461 VMFS partitions 464 vmkusage 588 vmstat command 545, 693, 699 VMware Consolidated Backup 460 VMware ESX Server See ESX Server VMware HA 460 VMX 83 vSwitch policy exceptions 467 VT 82 VTune 528<br />
<br />
W Web server Microsoft IIS 981–1016<br />
<br />
1064<br />
<br />
server type 19 stripe size 209 well balanced system 635 wide CPU architecture 60 Win32PrioritySeparation 304 Windows DisablePagingExecutive 976 Session Directory service 977 Windows 2000 Hyper-Threading 41 network monitor driver 502 Windows Management Instrumentation 522 Windows NT 4 GB, more than 151 Hyper-Threading 41 Windows Server 2003 295–370 /3GB parameter 334, 337 /PAE parameter 336 32-bit editions 297 4 GB memory limit 335 64-bit editions 297–298 8.3 filenames 359 active data set size 360 Active Directory 14 addressable memory 299 affinity 330 analyzing bottlenecks 655–685 auditing 362 auto-negotiation 325 AWE 154 AWE support 335 background services 304 Base Priority 328 binding order 320 checksum offload 259 clustering support 297 Coalesce Buffers setting 325 compression 758 CPU affinity 330 CPU-bound applications 330 CPUs supported 297 Data Execution Prevention 301 disable services 315 DisableLastAccess 755 DisablePagingExecutive 351, 355 DISKPERF command 668 drive arrays 361 dump file 307 duplex setting 325<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
dynamic priority 326 EM64T 301 EnablePMTUDiscovery 349 encryption 362 features 297 file server 745 file server performance 750 file system cache 309 foreground boost 303, 751 fragmentation 361 fsutil command 359 futures 367 high priority 329 Hyper-Threading 41 I/O locking operations 353 idle priority 329 Intel 64 Technology 301 interrupts 332 INTFILTR utility 332, 680 LanmanServer key 367 large amount of RAM installed 314 large TCP window scaling 340 LargeSystemCache 313 Last Access Time 358 link speed setting 325 log off the server 363 logical thread limit 370 Longhorn 368 MaxCmds 357 MaxFreeTcbs 346 MaxFreeTWTcbs 346 MaxHashTableSize 345 maximize throughput for file sharing 312 maximum segment size 339 Maximum Transmission Unit 347 MaxMpxCt 357 MaxUserPort 343 MaxWorkItems 357 monitoring tools 471 MTU 347 multitasking 303 NET SERVER CONFIG command 364 network card settings 322 network control blocks 357 Network Load Balancing 12 Network Monitor 511 network provider order 321 normal priority 329 NTFS<br />
<br />
tuning 756 NtfsDisable8dot3NameCreation 755 NumTcbTablePartitions 345, 755 offload features 326 outstanding network requests 356 packet segmentation 258 PAE support 335 page file stripe size 212 paged pool 300 PagedPoolSize 352, 754 paging 305, 307 Patch-Guard 301 Path MTU 349 performance bottlenecks 655–685 performance options window 304 performance tools 471 print provider order 321 priority 326 priority, when to change 330 processor affinity 330 product family 296 protocols binding order 320 remove 318 quantum 327 R2 302 RAM supported 297 realtime priority 329 Receive Buffers setting 325 receive window 339 registry parameters 754 resource kit 520 roadmap 368 Scalable Networking Pack 274 scheduler 326 screen savers 363 server roles 363 service startup recommendations 317 services, disable 315, 680 SMB 356 SNMP service 685 START command 328 Static Resource Allocation Table 103 stripe size 209, 212, 361 system cache 309, 751 SystemPages 353 Task Manager 327 See Task Manager TCP acknowledgement frequency 346<br />
<br />
Index<br />
<br />
1065<br />
<br />
TCP connection retransmissions 342 TCP Control Block table 344 TCP data retransmissions 342 TCP TIME-WAIT delay 343 TCP window scaling 340 TCP window size 339 TCP/IP operation 241 TCP/IP tuning 338 TCP1323Opts parameter 341 TcpAckFrequency 756 TcpWindowSize 339 Terminal Services See Windows Terminal Services TOE support 266 tools 471 TRACERPT 477 transfer size 362 Transmit Descriptors setting 326 user memory 299 virtual memory 299, 305, 663, 752 VTune 528 Win32PrioritySeparation 304 Windows on Windows 64 emulator 300 WMI 522 work items 357 x64 297 Windows Terminal Services 951–980 See also Citrix MetaFrame Active Session Limit 966 affinity to a network card 967 application server mode 962 applications 964 client session timeouts 966 counters 979 Data Collector 978 encryption 965 Idle Session Limit 966 L2 cache 23 Maximum Idle Time 966 memory 23 network 23 network card affinity 967 number of users 23 performance 23, 953 performance counters 979 processor 23 RDC 970 remote administration mode 962 resource mapping, disabling 968<br />
<br />
1066<br />
<br />
scalability 953 scale up/out 954 server type 22 Session Directory 977 subsystems 23 tuning 964 WINS servers 25 Woodcrest 48 word processors Performance console, use with 503 worksheets 634 write-back versus write-through 214<br />
<br />
X X3 Architecture cache associativity 71 memory implementation 158 scaling performance 73 XA-64e chipset 116 x3755 network performance 247 Passthru card 56 Xcelerated Memory Technology 164 x3950 memory implementation 160 XA-64e chipset 116 XceL4v 118 Xcelerated Memory Technology 164 xcopy, use of 257 Xen 82, 411, 420 binary translation 423 drivers 424 hypervisor layer 423 I/O virtualization 424 paravirtualization 421 performance 423 virtualization 420 XenSource 420 Xeon addressable memory 155 Xeon DP 42 Xeon MP 43 addressable memory 66 dual core 45 single core 36 xPL 561 command line 566 counter descriptions 562<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
CPU counters 562 CSV file 566 example 568 input.prm file 566 interrupts counters 564 log file 566 memory counters 565 network counters 566 parameter file 566–567 Performance Monitor, importing into 566 sample 568 settings 567 starting 566 stopping 566 xRef 100 xSeries See System x<br />
<br />
Y YaST 375<br />
<br />
Index<br />
<br />
1067<br />
<br />
1068<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
Tuning IBM System x Servers for Performance<br />
<br />
(1.5” spine) 1.5”<-> 1.998” 789 <->1051 pages<br />
<br />
Back cover<br />
<br />
®<br />
<br />
Tuning IBM System x Servers for Performance Identify and eliminate performance bottlenecks in key subsystems Expert knowledge from inside the IBM performance labs Covers Windows, Linux, and ESX Server<br />
<br />
This IBM Redbook describes what you can do to improve and maximize the performance of your business server applications running on IBM System x hardware and either Windows, Linux, or ESX Server operating systems. It describes how to improve the performance of the System x hardware, the operating system, and specific server applications.<br />
<br />
INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION<br />
<br />
The book is divided into five parts. Part 1 explains the technology implemented in the major subsystems in System x servers and shows what settings you can select or adjust to obtain the best performance. Part 2 describes the performance aspects of the operating systems: Microsoft Windows Server 2003, Red Hat Enterprise Linux, SUSE Linux Enterprise Server, and VMware ESX Server.<br />
<br />
BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE<br />
<br />
Part 3 introduces the performance monitoring tools that are available to users of System x servers. Part 4 shows how to analyze your system to find performance bottlenecks and what to do to eliminate them. Part 5 examines specific performance characteristics of specific server applications. This book is targeted at people who configure Intel and AMD processor-based servers that are running Windows, Linux, or ESX Server and seek to maximize performance. Some knowledge of servers is required. Skills in performance tuning are not assumed.<br />
<br />
SG24-5287-04<br />
<br />
ISBN 0738489794<br />
<br />
IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.<br />
<br />
For more information: ibm.com/redbooks<br />
<br />
</div>
</div>
</div>
<div class="row hidden-xs">
<div class="col-md-12">
<h2></h2>
<hr />
</div>
<div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/performance-tuning-for-linux-servers.html">
<img src="https://epdf.mx/img/300x300/performance-tuning-for-linux-servers_5a3c9715b7d7bcc32043ed84.jpg" alt="Performance Tuning for Linux Servers" />
<h3 class="note-title">Performance Tuning for Linux Servers</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/performance-tuning-for-linux-servers.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/system-performance-tuning.html">
<img src="https://epdf.mx/img/300x300/system-performance-tuning_5a842a0eb7d7bcc35d4017ca.jpg" alt="System Performance Tuning" />
<h3 class="note-title">System Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/system-performance-tuning.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/linux-on-ibm-system-z-performance-measurement-and-tuning.html">
<img src="https://epdf.mx/img/300x300/linux-on-ibm-system-z-performance-measurement-and-_5a83fc9fb7d7bcce4b4ff712.jpg" alt="Linux on IBM System Z: Performance Measurement and Tuning" />
<h3 class="note-title">Linux on IBM System Z: Performance Measurement and Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/linux-on-ibm-system-z-performance-measurement-and-tuning.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/ibm-certification-study-guide-aix-performance-and-system-tuning.html">
<img src="https://epdf.mx/img/300x300/ibm-certification-study-guide-aix-performance-and-_5a41565db7d7bc477ee771eb.jpg" alt="IBM Certification Study Guide AIX Performance and System Tuning" />
<h3 class="note-title">IBM Certification Study Guide AIX Performance and System Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/ibm-certification-study-guide-aix-performance-and-system-tuning.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/system-performance-tuning-nutshell-handbooks.html">
<img src="https://epdf.mx/img/300x300/system-performance-tuning-nutshell-handbooks_5a841ab5b7d7bcc62f17b05f.jpg" alt="System Performance Tuning (Nutshell Handbooks)" />
<h3 class="note-title">System Performance Tuning (Nutshell Handbooks)</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/system-performance-tuning-nutshell-handbooks.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/oracle-performance-tuning-for-10gr2.html">
<img src="https://epdf.mx/img/300x300/oracle-performance-tuning-for-10gr2_5b34785fb7d7bc3a7f7a7645.jpg" alt="Oracle Performance Tuning for 10gR2" />
<h3 class="note-title">Oracle Performance Tuning for 10gR2</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/oracle-performance-tuning-for-10gr2.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/informix-performance-tuning.html">
<img src="https://epdf.mx/img/300x300/informix-performance-tuning_5b94d613b7d7bc20653c98f6.jpg" alt="Informix Performance Tuning" />
<h3 class="note-title">Informix Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/informix-performance-tuning.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/java-performance-tuning24813.html">
<img src="https://epdf.mx/img/300x300/java-performance-tuning_5a6a636eb7d7bc4e72d2f293.jpg" alt="Java Performance Tuning" />
<h3 class="note-title">Java Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/java-performance-tuning24813.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/java-performance-tuning71849.html">
<img src="https://epdf.mx/img/300x300/java-performance-tuning_5a6a6379b7d7bc4e72d2f294.jpg" alt="Java Performance Tuning" />
<h3 class="note-title">Java Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/java-performance-tuning71849.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/java-performance-tuning.html">
<img src="https://epdf.mx/img/300x300/java-performance-tuning_5a6a6365b7d7bc4e72d2f292.jpg" alt="Java Performance Tuning" />
<h3 class="note-title">Java Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/java-performance-tuning.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/sybase-performance-tuning.html">
<img src="https://epdf.mx/img/300x300/sybase-performance-tuning_5ab5b07db7d7bcdf1f08de72.jpg" alt="Sybase Performance Tuning" />
<h3 class="note-title">Sybase Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/sybase-performance-tuning.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/sql-performance-tuning011e8d96b50f2e3154e7d08b48a6a94f89482.html">
<img src="https://epdf.mx/img/300x300/sql-performance-tuning_5b34780bb7d7bc397f68fee7.jpg" alt="SQL Performance Tuning" />
<h3 class="note-title">SQL Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/sql-performance-tuning011e8d96b50f2e3154e7d08b48a6a94f89482.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/sql-performance-tuninga9c94e1740efb7da996ec5e8aa53274e42704.html">
<img src="https://epdf.mx/img/300x300/sql-performance-tuning_5b94d62bb7d7bc1f65c9f6e1.jpg" alt="SQL Performance Tuning" />
<h3 class="note-title">SQL Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/sql-performance-tuninga9c94e1740efb7da996ec5e8aa53274e42704.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/sql-performance-tuning.html">
<img src="https://epdf.mx/img/300x300/sql-performance-tuning_5a725fd2b7d7bc4b2d6bcc23.jpg" alt="SQL Performance Tuning" />
<h3 class="note-title">SQL Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/sql-performance-tuning.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/sendmail-performance-tuning.html">
<img src="https://epdf.mx/img/300x300/sendmail-performance-tuning_5a725f8fb7d7bc4b2d6bcc20.jpg" alt="sendmail Performance Tuning" />
<h3 class="note-title">sendmail Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/sendmail-performance-tuning.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/oracle-performance-tuning-and-optimization.html">
<img src="https://epdf.mx/img/300x300/oracle-performance-tuning-and-optimization_5a8de511b7d7bccf55c21393.jpg" alt="Oracle Performance Tuning and Optimization" />
<h3 class="note-title">Oracle Performance Tuning and Optimization</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/oracle-performance-tuning-and-optimization.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/oracle-performance-tuning-nutshell-handbooks.html">
<img src="https://epdf.mx/img/300x300/oracle-performance-tuning-nutshell-handbooks_5ac7ed24b7d7bcf76247bacc.jpg" alt="Oracle Performance Tuning (Nutshell Handbooks)" />
<h3 class="note-title">Oracle Performance Tuning (Nutshell Handbooks)</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/oracle-performance-tuning-nutshell-handbooks.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/java-performance-tuning-2nd-edition.html">
<img src="https://epdf.mx/img/300x300/java-performance-tuning-2nd-edition_5a6a631ab7d7bc4f7273ab57.jpg" alt="Java Performance Tuning (2nd Edition)" />
<h3 class="note-title">Java Performance Tuning (2nd Edition)</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/java-performance-tuning-2nd-edition.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/jboss-as-5-performance-tuning.html">
<img src="https://epdf.mx/img/300x300/jboss-as-5-performance-tuning_5ab6f50db7d7bc9e1f81eba0.jpg" alt="JBoss AS 5 Performance Tuning" />
<h3 class="note-title">JBoss AS 5 Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/jboss-as-5-performance-tuning.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/ibm-system-storage-solutions-handbook.html">
<img src="https://epdf.mx/img/300x300/ibm-system-storage-solutions-handbook_5a7e4018b7d7bcd64bd2b337.jpg" alt="IBM System Storage Solutions Handbook" />
<h3 class="note-title">IBM System Storage Solutions Handbook</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/ibm-system-storage-solutions-handbook.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/oracle-database-11g-r2-performance-tuning-cookbook.html">
<img src="https://epdf.mx/img/300x300/oracle-database-11g-r2-performance-tuning-cookbook_5b5f74c4b7d7bc5e37200ca7.jpg" alt="Oracle Database 11g R2 Performance Tuning Cookbook" />
<h3 class="note-title">Oracle Database 11g R2 Performance Tuning Cookbook</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/oracle-database-11g-r2-performance-tuning-cookbook.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/pro-ios-apps-performance-optimization-and-tuning98124.html">
<img src="https://epdf.mx/img/300x300/pro-ios-apps-performance-optimization-and-tuning_5a483c52b7d7bcab0322bd57.jpg" alt="Pro IOS Apps Performance Optimization and Tuning" />
<h3 class="note-title">Pro IOS Apps Performance Optimization and Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/pro-ios-apps-performance-optimization-and-tuning98124.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/maximizing-performance-and-scalability-with-ibm-websphere.html">
<img src="https://epdf.mx/img/300x300/maximizing-performance-and-scalability-with-ibm-we_5a88c686b7d7bc9f476ed242.jpg" alt="Maximizing Performance and Scalability with IBM WebSphere" />
<h3 class="note-title">Maximizing Performance and Scalability with IBM WebSphere</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/maximizing-performance-and-scalability-with-ibm-websphere.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/ocp-oracle9i-performance-tuning-study-guide.html">
<img src="https://epdf.mx/img/300x300/ocp-oracle9i-performance-tuning-study-guide_5b347826b7d7bc3c7f8d46dc.jpg" alt="OCP: Oracle9i performance tuning study guide" />
<h3 class="note-title">OCP: Oracle9i performance tuning study guide</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/ocp-oracle9i-performance-tuning-study-guide.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/linux-performance-tuning-and-capacity-planning-kaleidoscope.html">
<img src="https://epdf.mx/img/300x300/linux-performance-tuning-and-capacity-planning-kal_5a8400deb7d7bccd4bb6ae8a.jpg" alt="Linux Performance Tuning and Capacity Planning (Kaleidoscope)" />
<h3 class="note-title">Linux Performance Tuning and Capacity Planning (Kaleidoscope)</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/linux-performance-tuning-and-capacity-planning-kaleidoscope.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/professional-sql-server-2005-performance-tuning.html">
<img src="https://epdf.mx/img/300x300/professional-sql-server-2005-performance-tuning_5a75475bb7d7bc8315530f76.jpg" alt="Professional SQL Server 2005 Performance Tuning" />
<h3 class="note-title">Professional SQL Server 2005 Performance Tuning</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/professional-sql-server-2005-performance-tuning.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/sql-server-2008-query-performance-tuning-distilled4d62ea326f99c92ab2d93bcde24d1ea811390.html">
<img src="https://epdf.mx/img/300x300/sql-server-2008-query-performance-tuning-distilled_5a7547e8b7d7bc82151b5c68.jpg" alt="SQL Server 2008 Query Performance Tuning Distilled" />
<h3 class="note-title">SQL Server 2008 Query Performance Tuning Distilled</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/sql-server-2008-query-performance-tuning-distilled4d62ea326f99c92ab2d93bcde24d1ea811390.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/performance-tuning-and-optimizing-aspnet-applications.html">
<img src="https://epdf.mx/img/300x300/performance-tuning-and-optimizing-aspnet-applicati_5a5563b5b7d7bc515ebc9cd6.jpg" alt="Performance Tuning and Optimizing ASP.NET Applications" />
<h3 class="note-title">Performance Tuning and Optimizing ASP.NET Applications</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/performance-tuning-and-optimizing-aspnet-applications.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/oracle-database-11g-r2-performance-tuning-cookbook05ed66f5ce06228d93ff56b6e6942e8856164.html">
<img src="https://epdf.mx/img/300x300/oracle-database-11g-r2-performance-tuning-cookbook_5b5f74cdb7d7bc6137283e2c.jpg" alt="Oracle Database 11g R2 Performance Tuning Cookbook" />
<h3 class="note-title">Oracle Database 11g R2 Performance Tuning Cookbook</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/oracle-database-11g-r2-performance-tuning-cookbook05ed66f5ce06228d93ff56b6e6942e8856164.html">Read more</a>
</div>
</div>
</div> <div class="col-lg-2 col-md-3">
<div class="note">
<div class="note-meta-thumb">
<a href="https://epdf.mx/ocp-oracle9i-performance-tuning-study-guidedc952330bc055aca78cea4c8aa3e79e596647.html">
<img src="https://epdf.mx/img/300x300/ocp-oracle9i-performance-tuning-study-guide_5b347827b7d7bc397f68fee8.jpg" alt="OCP: Oracle9i performance tuning study guide" />
<h3 class="note-title">OCP: Oracle9i performance tuning study guide</h3>
</a>
</div>
<div class="note-action">
<a class="more-link" href="https://epdf.mx/ocp-oracle9i-performance-tuning-study-guidedc952330bc055aca78cea4c8aa3e79e596647.html">Read more</a>
</div>
</div>
</div> </div>
</div>
<div class="col-lg-3 col-md-4 col-xs-12">
<div class="panel-recommend panel panel-primary">
<div class="panel-heading">
<h4 class="panel-title">Recommend Documents</h4>
</div>
<div class="panel-body">
<div class="row m-0">
<div class="col-md-3 col-xs-3 pl-0 text-center">
<a href="https://epdf.mx/performance-tuning-for-linux-servers.html">
<img src="https://epdf.mx/img/60x80/performance-tuning-for-linux-servers_5a3c9715b7d7bcc32043ed84.jpg" alt="" width="100%" />
</a>
</div>
<div class="col-md-9 col-xs-9 p-0">
<label>
<a href="https://epdf.mx/performance-tuning-for-linux-servers.html">
Performance Tuning for Linux Servers </a>
</label>
<div class="note-meta">
<div class="note-desc"></div>
</div>
</div>
<div class="clearfix"></div>
<hr class="mt-15 mb-15" />
</div> <div class="row m-0">
<div class="col-md-3 col-xs-3 pl-0 text-center">
<a href="https://epdf.mx/system-performance-tuning.html">
<img src="https://epdf.mx/img/60x80/system-performance-tuning_5a842a0eb7d7bcc35d4017ca.jpg" alt="" width="100%" />
</a>
</div>
<div class="col-md-9 col-xs-9 p-0">
<label>
<a href="https://epdf.mx/system-performance-tuning.html">
System Performance Tuning </a>
</label>
<div class="note-meta">
<div class="note-desc"></div>
</div>
</div>
<div class="clearfix"></div>
<hr class="mt-15 mb-15" />
</div> <div class="row m-0">
<div class="col-md-3 col-xs-3 pl-0 text-center">
<a href="https://epdf.mx/linux-on-ibm-system-z-performance-measurement-and-tuning.html">
<img src="https://epdf.mx/img/60x80/linux-on-ibm-system-z-performance-measurement-and-_5a83fc9fb7d7bcce4b4ff712.jpg" alt="" width="100%" />
</a>
</div>
<div class="col-md-9 col-xs-9 p-0">
<label>
<a href="https://epdf.mx/linux-on-ibm-system-z-performance-measurement-and-tuning.html">
Linux on IBM System Z: Performance Measurement and Tuning </a>
</label>
<div class="note-meta">
<div class="note-desc">Front cover
Linux on IBM System z: Performance Measurement and Tuning Understanding Linux performance on System z z/VM ...</div>
</div>
</div>
<div class="clearfix"></div>
<hr class="mt-15 mb-15" />
</div> <div class="row m-0">
<div class="col-md-3 col-xs-3 pl-0 text-center">
<a href="https://epdf.mx/ibm-certification-study-guide-aix-performance-and-system-tuning.html">
<img src="https://epdf.mx/img/60x80/ibm-certification-study-guide-aix-performance-and-_5a41565db7d7bc477ee771eb.jpg" alt="" width="100%" />
</a>
</div>
<div class="col-md-9 col-xs-9 p-0">
<label>
<a href="https://epdf.mx/ibm-certification-study-guide-aix-performance-and-system-tuning.html">
IBM Certification Study Guide AIX Performance and System Tuning </a>
</label>
<div class="note-meta">
<div class="note-desc">IBM Certification Study Guide AIX Performance and System Tuning Developed specifically for the purpose of preparing for ...</div>
</div>
</div>
<div class="clearfix"></div>
<hr class="mt-15 mb-15" />
</div> <div class="row m-0">
<div class="col-md-3 col-xs-3 pl-0 text-center">
<a href="https://epdf.mx/system-performance-tuning-nutshell-handbooks.html">
<img src="https://epdf.mx/img/60x80/system-performance-tuning-nutshell-handbooks_5a841ab5b7d7bcc62f17b05f.jpg" alt="" width="100%" />
</a>
</div>
<div class="col-md-9 col-xs-9 p-0">
<label>
<a href="https://epdf.mx/system-performance-tuning-nutshell-handbooks.html">
System Performance Tuning (Nutshell Handbooks) </a>
</label>
<div class="note-meta">
<div class="note-desc"></div>
</div>
</div>
<div class="clearfix"></div>
<hr class="mt-15 mb-15" />
</div> <div class="row m-0">
<div class="col-md-3 col-xs-3 pl-0 text-center">
<a href="https://epdf.mx/oracle-performance-tuning-for-10gr2.html">
<img src="https://epdf.mx/img/60x80/oracle-performance-tuning-for-10gr2_5b34785fb7d7bc3a7f7a7645.jpg" alt="" width="100%" />
</a>
</div>
<div class="col-md-9 col-xs-9 p-0">
<label>
<a href="https://epdf.mx/oracle-performance-tuning-for-10gr2.html">
Oracle Performance Tuning for 10gR2 </a>
</label>
<div class="note-meta">
<div class="note-desc">Oracle® Performance Tuning for 10gR2 Second Edition
Oracle Database Related Book Titles:
Oracle 9iR2 Data Warehousin...</div>
</div>
</div>
<div class="clearfix"></div>
<hr class="mt-15 mb-15" />
</div> <div class="row m-0">
<div class="col-md-3 col-xs-3 pl-0 text-center">
<a href="https://epdf.mx/informix-performance-tuning.html">
<img src="https://epdf.mx/img/60x80/informix-performance-tuning_5b94d613b7d7bc20653c98f6.jpg" alt="" width="100%" />
</a>
</div>
<div class="col-md-9 col-xs-9 p-0">
<label>
<a href="https://epdf.mx/informix-performance-tuning.html">
Informix Performance Tuning </a>
</label>
<div class="note-meta">
<div class="note-desc"></div>
</div>
</div>
<div class="clearfix"></div>
<hr class="mt-15 mb-15" />
</div> <div class="row m-0">
<div class="col-md-3 col-xs-3 pl-0 text-center">
<a href="https://epdf.mx/java-performance-tuning24813.html">
<img src="https://epdf.mx/img/60x80/java-performance-tuning_5a6a636eb7d7bc4e72d2f293.jpg" alt="" width="100%" />
</a>
</div>
<div class="col-md-9 col-xs-9 p-0">
<label>
<a href="https://epdf.mx/java-performance-tuning24813.html">
Java Performance Tuning </a>
</label>
<div class="note-meta">
<div class="note-desc"></div>
</div>
</div>
<div class="clearfix"></div>
<hr class="mt-15 mb-15" />
</div> <div class="row m-0">
<div class="col-md-3 col-xs-3 pl-0 text-center">
<a href="https://epdf.mx/java-performance-tuning71849.html">
<img src="https://epdf.mx/img/60x80/java-performance-tuning_5a6a6379b7d7bc4e72d2f294.jpg" alt="" width="100%" />
</a>
</div>
<div class="col-md-9 col-xs-9 p-0">
<label>
<a href="https://epdf.mx/java-performance-tuning71849.html">
Java Performance Tuning </a>
</label>
<div class="note-meta">
<div class="note-desc">O’reilly - Java Performance Tuning Java Performance Tuning Copyright © 2000 O'Reilly & Associates, Inc. All rights rese...</div>
</div>
</div>
<div class="clearfix"></div>
<hr class="mt-15 mb-15" />
</div> <div class="row m-0">
<div class="col-md-3 col-xs-3 pl-0 text-center">
<a href="https://epdf.mx/java-performance-tuning.html">
<img src="https://epdf.mx/img/60x80/java-performance-tuning_5a6a6365b7d7bc4e72d2f292.jpg" alt="" width="100%" />
</a>
</div>
<div class="col-md-9 col-xs-9 p-0">
<label>
<a href="https://epdf.mx/java-performance-tuning.html">
Java Performance Tuning </a>
</label>
<div class="note-meta">
<div class="note-desc">Team[oR] 2001 [x] java
O’reilly - Java Performance Tuning Java Performance Tuning Copyright © 2000 O'Reilly & Associat...</div>
</div>
</div>
<div class="clearfix"></div>
<hr class="mt-15 mb-15" />
</div> </div>
</div>
</div>
</div>
</div>
<div class="modal fade" id="report" tabindex="-1" role="dialog" aria-hidden="true">
<div class="modal-dialog">
<div class="modal-content">
<form role="form" method="post" action="https://epdf.mx/report/tuning-ibm-system-x-servers-for-performance" style="border: none;">
<div class="modal-header">
<button type="button" class="close" data-dismiss="modal" aria-hidden="true">×</button>
<h4 class="modal-title">Report "Tuning IBM System X Servers for Performance"</h4>
</div>
<div class="modal-body">
<div class="form-group">
<label>Your name</label>
<input type="text" name="name" required="required" class="form-control" />
</div>
<div class="form-group">
<label>Email</label>
<input type="email" name="email" required="required" class="form-control" />
</div>
<div class="form-group">
<label>Reason</label>
<select name="reason" required="required" class="form-control">
<option value="">-Select Reason-</option>
<option value="pornographic" selected="selected">Pornographic</option>
<option value="defamatory">Defamatory</option>
<option value="illegal">Illegal/Unlawful</option>
<option value="spam">Spam</option>
<option value="others">Other Terms Of Service Violation</option>
<option value="copyright">File a copyright complaint</option>
</select>
</div>
<div class="form-group">
<label>Description</label>
<textarea name="description" required="required" rows="3" class="form-control" style="border: 1px solid #cccccc;"></textarea>
</div>
<div class="form-group">
<div style="display: inline-block;">
<div class="g-recaptcha" data-sitekey="6Le0e5ceAAAAADsZpn1H3VI-HvOppGDh-O-QAVYL"></div>
</div>
</div>
<script src='https://www.google.com/recaptcha/api.js'></script>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
<button type="submit" class="btn btn-success">Send</button>
</div>
</form>
</div>
</div>
</div>
<footer class="footer" style="margin-top: 60px;">
<div class="container-fluid">
Copyright © 2024 EPDF.MX. All rights reserved.
<div class="pull-right">
<span><a href="https://epdf.mx/about">About Us</a></span> |
<span><a href="https://epdf.mx/privacy">Privacy Policy</a></span> |
<span><a href="https://epdf.mx/term">Terms of Service</a></span> |
<span><a href="https://epdf.mx/copyright">Copyright</a></span> |
<span><a href="https://epdf.mx/dmca">DMCA</a></span> |
<span><a href="https://epdf.mx/contact">Contact Us</a></span> |
<span><a href="https://epdf.mx/cookie_policy">Cookie Policy</a></span>
</div>
</div>
</footer>
<!-- Modal -->
<div class="modal fade" id="login" tabindex="-1" role="dialog" aria-labelledby="myModalLabel">
<div class="modal-dialog" role="document">
<div class="modal-content">
<div class="modal-header">
<button type="button" class="close" data-dismiss="modal" aria-label="Close" on="tap:login.close"><span aria-hidden="true">×</span></button>
<h4 class="modal-title" id="add-note-label">Sign In</h4>
</div>
<div class="modal-body">
<form action="https://epdf.mx/login" method="post">
<div class="form-group">
<label class="sr-only" for="email">Email</label>
<input class="form-input form-control" type="text" name="email" id="email" value="" placeholder="Email" />
</div>
<div class="form-group">
<label class="sr-only" for="password">Password</label>
<input class="form-input form-control" type="password" name="password" id="password" value="" placeholder="Password" />
</div>
<div class="form-group">
<div class="checkbox">
<label class="form-checkbox">
<input type="checkbox" name="remember" value="1" />
<i class="form-icon"></i> Remember me
</label>
<label class="pull-right"><a href="https://epdf.mx/forgot">Forgot password?</a></label>
</div>
</div>
<button class="btn btn-primary btn-block" type="submit">Sign In</button>
</form>
<hr style="margin-top: 15px;" />
<a href="https://epdf.mx/login/facebook" class="btn btn-facebook btn-block"><i class="fa fa-facebook"></i> Login with Facebook</a>
</div>
</div>
</div>
</div>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-111550345-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-111550345-1');
</script>
<script src="https://epdf.mx/assets/js/jquery-ui.min.js"></script>
<link rel="stylesheet" href="https://epdf.mx/assets/css/jquery-ui.css">
<script>
$(function () {
$("#document_search").autocomplete({
source: function (request, response) {
$.ajax({
url: "https://epdf.mx/suggest",
dataType: "json",
data: {
term: request.term
},
success: function (data) {
response(data);
}
});
},
autoFill: true,
select: function( event, ui ) {
$(this).val(ui.item.value);
$(this).parents("form").submit();
}
});
});
</script>
<!-- cookie policy -->
<div id="EPDFMX_cookie_box" style="z-index:99999; border-top: 1px solid #fefefe; background: #97c479; width: 100%; position: fixed; padding: 5px 15px; text-align: center; left:0; bottom: 0;">
Our partners will collect data and use cookies for ad personalization and measurement. <a href="https://epdf.mx/cookie_policy" target="_blank">Learn how we and our ad partner Google, collect and use data</a>. <a href="#" class="btn btn-success" onclick="accept_EPDFMX_cookie_box();return false;">Agree & close</a>
</div>
<script>
function accept_EPDFMX_cookie_box() {
document.cookie = "EPDFMX_cookie_box_viewed=1;max-age=15768000;path=/";
hide_EPDFMX_cookie_box();
}
function hide_EPDFMX_cookie_box() {
var cb = document.getElementById('EPDFMX_cookie_box');
if (cb) {
cb.parentElement.removeChild(cb);
}
}
(function () {
var EPDFMX_cookie_box_viewed = (function (name) {
var matches = document.cookie.match(new RegExp("(?:^|; )" + name.replace(/([\.$?*|{}\(\)\[\]\\\/\+^])/g, '\\$1') + "=([^;]*)"));
return matches ? decodeURIComponent(matches[1]) : undefined;
})('EPDFMX_cookie_box_viewed');
if (EPDFMX_cookie_box_viewed) {
hide_EPDFMX_cookie_box();
}
})();
</script>
<!-- end cookie policy -->
</body>
</html>
<script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script>