Front cover
Implementing the IBM System Storage SAN Volume Controller V4.3 Install, use, and troubleshoot the SAN Volume Controller Learn how to implement block virtualization Create space-efficient VDisks
Jon Tate Sameer Dhulekar Juerg Hossli Dan Koeck Suad Musovich
ibm.com/redbooks
International Technical Support Organization Implementing the IBM System Storage SAN Volume Controller V4.3 October 2008
SG24-6423-06
Note: Before using this information and the product it supports, read the information in “Notices” on page xv.
Seventh Edition (October 2008) This edition applies to Version 4 Release 3 of the IBM System Storage SAN Volume Controller. © Copyright International Business Machines Corporation 2003-2008. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
Contents Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii The team that wrote this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Summary of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii October 2008, Seventh Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii Chapter 1. Introduction to storage virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 The need for storage virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 In-band virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Out-of-band virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 IBM Global Parallel File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1 2 3 6 7 8
Chapter 2. IBM System Storage SAN Volume Controller overview . . . . . . . . . . . . . . . . 9 2.1 Maximum supported configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2 Glossary of commonly used terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 Virtualization overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.4 Compass architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.4.1 SAN Volume Controller clustering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4.2 SAN Volume Controller virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.4.3 SAN Volume Controller multipathing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.4.4 SAN Volume Controller logical configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.5 Software licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.6 New with SVC V4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.6.1 Space-efficient VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.6.2 FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.6.3 VDisk mirroring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.6.4 IPv6 addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.6.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.6.6 Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.6.7 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.6.8 Additional interoperability support since SVC 4.2.1 . . . . . . . . . . . . . . . . . . . . . . . 22 2.6.9 SVC 4.3 Interoperability enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Chapter 3. Planning and configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 General planning rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Physical planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Preparing your UPS environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Physical rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Cable connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 SAN planning and configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 SAN definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Fibre Channel switches, fabrics, interswitch links, and hops . . . . . . . . . . . . . . . . 3.3.3 General design considerations with the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . .
© Copyright IBM Corp. 2003-2008. All rights reserved.
25 26 27 29 30 31 32 33 35 35
iii
3.3.4 Boot support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.5 Configuration saving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.6 High availability SAN design and configuration rules with SVC . . . . . . . . . . . . . . 3.4 Zoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.1 Dual room high availability configuration with the SVC. . . . . . . . . . . . . . . . . . . . . 3.5.2 Local and remote SAN fabrics with SVC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.3 Technologies for extending the distance between two SVC clusters . . . . . . . . . . 3.6 SVC disk subsystem planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.1 Block virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.2 MDGs, I/O groups, virtual disks, and managed disks . . . . . . . . . . . . . . . . . . . . . . 3.6.3 Extents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.4 Image mode virtual disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.5 Managed mode virtual disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.6 Space-efficient Virtual Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.7 Extent allocation and size rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.8 MDisk group planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.9 Planning a virtual disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.10 Planning for operations on virtual disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.11 Host considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.12 Quorum disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.13 Expanding an SVC cluster configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Storage subsystem planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.1 Adding DS8000 storage to the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.2 Adding DS4000 storage to the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.3 LUN layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39 39 40 40 41 44 44 45 46 46 46 48 48 49 52 55 55 56 59 61 62 64 65 67 72 76
Chapter 4. Performance and capacity planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Performance considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 SAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 Disk subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.3 SVC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.4 Host. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Performance modeling and sizing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Performance monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Collecting performance statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Cluster wide statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Per node statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
77 78 78 79 80 87 88 88 88 88 90
Chapter 5. SVC Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.1 Systems Storage Productivity Center overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 5.1.1 SSPC hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 5.1.2 Example hardware configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.2 SVC Console software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.3 Installation planning information for the SSPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.4 Secure Shell overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.4.1 Generating public and private SSH key pairs using PuTTY . . . . . . . . . . . . . . . . . 98 5.5 Basic installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 5.5.1 Creating the cluster (first time) using the service panel . . . . . . . . . . . . . . . . . . . 102 5.6 Completing the initial cluster setup using the SAN Volume Controller Console GUI . 106 5.6.1 Configuring the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 5.6.2 Uploading the SSH public key to the SVC cluster. . . . . . . . . . . . . . . . . . . . . . . . 115 5.6.3 Uploading SSH public key(s) sample scenarios . . . . . . . . . . . . . . . . . . . . . . . . . 121
iv
Implementing the IBM System Storage SAN Volume Controller V4.3
5.6.4 Configuring the PuTTY session for the CLI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.5 Starting the PuTTY CLI session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 Using IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.1 Migrating a cluster from IPv4 to IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.2 Migrating a cluster from IPv6 to IPv4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8 Upgrading the SVC Console software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9 E-mail error notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
123 127 131 131 141 141 146
Chapter 6. Quickstart configuration using the command-line interface . . . . . . . . . . 6.1 Adding nodes to the cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Setting the cluster time zone and time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Checking the license features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Creating host definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Displaying managed disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Creating managed disk groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Creating a virtual disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.8 Assigning a VDisk to a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
157 158 161 162 162 164 165 167 171
Chapter 7. Quickstart configuration using the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Adding nodes to the cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.1 Installing certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Setting the cluster time zone and time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Checking the license status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Creating host definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Displaying managed disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Creating managed disk groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Creating a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7.1 Creating a space-efficient VDisk (SEV Disk) . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7.2 Creating a mirrored VDisk. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8 Assigning a VDisk to a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
173 174 178 183 184 185 187 188 192 198 203 207
Chapter 8. Host configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 SVC setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.1 Switch zoning recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.2 Using port masking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 AIX-specific information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 Configuring the AIX host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.2 Operating system versions and maintenance levels. . . . . . . . . . . . . . . . . . . . . . 8.2.3 HBAs for IBM System p hosts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.4 Configuring for fast fail and dynamic tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.5 Subsystem Device Driver (SDDPCM or SDD) . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.6 Discovering the assigned VDisk using SDD and AIX 5L V5.3 . . . . . . . . . . . . . . 8.2.7 Using SDD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.8 Creating and preparing volumes for use with AIX 5L V5.3 and SDD . . . . . . . . . 8.2.9 Discovering the assigned VDisk using AIX V6.1 and SDDPCM . . . . . . . . . . . . . 8.2.10 Using SDDPCM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.11 Creating and preparing volumes for use with AIX V6.1 and SDDPCM. . . . . . . 8.2.12 Expanding an AIX volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.13 Removing an SVC volume on AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.14 Running SVC commands from an AIX host system . . . . . . . . . . . . . . . . . . . . . 8.3 Windows-specific information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.1 Configuring Windows 2000, Windows 2003, and Windows 2008 hosts . . . . . . . 8.3.2 Configuring Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.3 Hardware lists, device driver, HBAs and firmware levels . . . . . . . . . . . . . . . . . .
209 210 211 212 212 213 213 213 214 215 218 222 223 223 227 228 228 232 232 233 233 233 233
Contents
v
8.3.4 Host adapter installation and configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.5 Changing the disk timeout on Microsoft Windows Server. . . . . . . . . . . . . . . . . . 8.3.6 SDD driver installation on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.7 SDDDSM driver installation on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Discovering the assigned VDisk in Windows 2000 / 2003 . . . . . . . . . . . . . . . . . . . . . 8.4.1 Extending a Windows 2000 or 2003 volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Example configuration of attaching an SVC to a Windows 2008 host . . . . . . . . . . . . 8.5.1 Installing SDDDSM on a Windows 2008 host . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.2 Installing SDDDSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.3 Attaching SVC VDisks to Windows 2008 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.4 Extending a Windows 2008 Volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.5 Removing a disk on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Using the SVC CLI from a Windows host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 Microsoft Volume Shadow Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7.1 Installation overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7.2 System requirements for the IBM System Storage hardware provider . . . . . . . . 8.7.3 Installing the IBM System Storage hardware provider . . . . . . . . . . . . . . . . . . . . 8.7.4 Verifying the installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7.5 Creating the free and reserved pools of volumes . . . . . . . . . . . . . . . . . . . . . . . . 8.7.6 Changing the configuration parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8 Linux (on Intel) specific information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8.1 Configuring the Linux host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8.2 Configuration information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8.3 Disabling automatic Linux system updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8.4 Setting queue depth with QLogic HBAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8.5 Multipathing in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8.6 Creating and preparing SDD volumes for use . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8.7 Using the operating system MPIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8.8 Creating and preparing MPIO volumes for use. . . . . . . . . . . . . . . . . . . . . . . . . . 8.9 VMware configuration information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.1 Configuring VMware hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.2 Operating system versions and maintenance levels. . . . . . . . . . . . . . . . . . . . . . 8.9.3 Guest operating systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.4 HBAs for hosts running VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.5 Multipath solutions supported . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.6 VMware storage and zoning recommendations . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.7 Setting the HBA timeout for failover in VMware . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.8 Multipathing in ESX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.9 Attaching VMware to VDisks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.10 VDisk naming in VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.11 Setting the Microsoft guest operating system timeout . . . . . . . . . . . . . . . . . . . 8.9.12 Extending a VMFS volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.13 Removing a datastore from an ESX host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.10 SUN Solaris support information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.10.1 Operating system versions and maintenance levels. . . . . . . . . . . . . . . . . . . . . 8.10.2 SDD dynamic pathing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.11 HP-UX configuration information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.11.1 Operating system versions and maintenance levels. . . . . . . . . . . . . . . . . . . . . 8.11.2 Multipath solutions supported . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.11.3 Co-existence of SDD and PV Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.11.4 Using an SVC VDisk as a cluster lock disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.11.5 Support for HP-UX greater than eight LUNs . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.12 Using SDDDSM, SDDPCM, and SDD Web interface . . . . . . . . . . . . . . . . . . . . . . . . vi
Implementing the IBM System Storage SAN Volume Controller V4.3
234 236 236 238 240 244 249 249 252 254 260 260 263 264 265 265 265 269 270 271 274 274 274 274 275 275 280 282 282 287 287 287 287 287 288 289 290 291 291 294 295 295 297 298 298 298 299 299 299 299 300 300 300
8.13 Calculating the queue depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 8.14 Further sources of information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 Chapter 9. SVC configuration and administration using the CLI . . . . . . . . . . . . . . . . 9.1 Managing users using the CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.1 Maintaining SSH keys using the CLI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.2 Managing user roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Managing the cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.1 Organizing on screen content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.2 Viewing cluster properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.3 Changing cluster settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.4 Maintaining cluster passwords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.5 Modifying IP addresses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.6 Setting the cluster time zone and time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.7 Start statistics collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.8 Stopping a statistics collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.9 Audit Log commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.10 Status of discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.11 Status of copy operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.12 Shutting down a cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Working with nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.1 I/O groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.2 Viewing I/O group details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.3 Renaming an I/O group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.4 Adding and removing hostiogrp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.5 Listing I/O groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Viewing node details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.2 Adding a node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.3 Renaming a node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.4 Deleting a node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.5 Shutting down a node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 Working with managed disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.1 Disk controller systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.2 MDisk information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.3 Renaming an MDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.4 Discovering MDisks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.5 Setting up a quorum disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.6 Including an MDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.7 Showing the MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.8 Showing a VDisk on an MDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Managed Disk Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6.1 Creating an MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6.2 Renaming an MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6.3 Deleting an MDisk group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6.4 Adding MDisks to an MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6.5 Removing MDisks from MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6.6 Showing MDisks in a MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6.7 Showing VDisks using a MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.1 Host information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.2 Creating a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.3 Modify a host. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents
303 304 304 305 307 308 309 311 314 314 315 316 317 318 319 319 320 321 321 321 321 322 323 323 323 324 325 326 326 327 327 328 329 330 330 332 333 333 334 334 335 336 336 336 337 337 338 338 338 340
vii
viii
9.7.4 Deleting a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.5 Adding ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.6 Deleting ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.8 SAN debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9 Working with virtual disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.1 VDisk information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.2 Creating a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.3 Creating a VDisk in image mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.4 Adding a mirrored VDisk copy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.5 Splitting a VDisk Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.6 Deleting a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.7 Expanding a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.8 Mapping a VDisk to a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.9 Deleting a VDisk-to-host mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.10 Showing the VDisks mapped to a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.11 Modifying a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.12 Migrating a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.13 Migrate a VDisk to an image mode VDisk. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.14 Shrinking a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.15 Showing the MDisks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.16 Showing the MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9.17 Showing the host to which the VDisk is mapped . . . . . . . . . . . . . . . . . . . . . . . 9.9.18 Showing the VDisk to which the host is mapped . . . . . . . . . . . . . . . . . . . . . . . 9.9.19 Tracing a host disk back to its source physical disk . . . . . . . . . . . . . . . . . . . . . 9.10 Service and maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.1 Upgrading software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.2 Running maintenance procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.3 Setting up error notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.4 Analyzing the error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.5 License settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10.6 Viewing the feature log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.11 SVC cluster configuration backup and recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.11.1 Backing up the SVC cluster configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.11.2 Restoring the SVC cluster configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.11.3 Deleting configuration backup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.12 Listing dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.12.1 Error or event dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.12.2 Featurization log dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.12.3 I/O trace dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.12.4 I/O statistics dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.12.5 Software dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.12.6 Application abends dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.12.7 Other node dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.13 T3 recovery process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.14 Scripting and its usage under CLI for SVC task automation . . . . . . . . . . . . . . . . . . .
341 341 343 344 346 346 348 349 351 357 358 359 361 362 362 363 367 368 369 371 372 373 373 374 375 375 381 384 385 387 388 389 390 393 393 394 394 395 395 395 396 396 397 399 399
Chapter 10. SVC configuration and administration using the GUI. . . . . . . . . . . . . . . 10.1 Managing users. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1.1 Creating a user using the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1.2 Modifying a user role. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1.3 Deleting a user role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Managing the cluster using the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.1 Organizing on screen content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
401 402 402 404 406 407 407
Implementing the IBM System Storage SAN Volume Controller V4.3
10.2.2 Viewing cluster properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.3 Maintain cluster passwords. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.4 Modifying IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.5 Setting the cluster time zone and time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.6 Starting the statistics collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.7 Stopping the statistics collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.8 Shutting down a cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Working with nodes using the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.1 I/O groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.2 Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 Viewing progress. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5 Working with managed disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.1 Disk controller systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.2 Discovery status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.3 Managed disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.4 MDisk information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.5 Renaming an MDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.6 Discovering MDisks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.7 Setting up a quorum disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.8 Including an MDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.9 Showing an MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.10 Showing a VDisk for an MDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.11 Creating a VDisk in image mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.12 Creating an image mode mirrored VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6 Managed disk groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.1 Viewing MDisk group information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.2 Creating an MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.3 Renaming an MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.4 Deleting an MDisk group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.5 Adding MDisks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.6 Removing MDisks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.7 Showing MDisks in this group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6.8 Showing VDisks using this group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7 Working with hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.1 Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.2 Host information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.3 Creating a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.4 Modifying a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.5 Deleting a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.6 Adding ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.7 Deleting ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.8 Fabrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8 Working with virtual disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.1 Using the Virtual Disks window for VDisks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.2 VDisk information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.3 Creating a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.4 Creating a space-efficient VDisk with auto-expand. . . . . . . . . . . . . . . . . . . . . . 10.8.5 Deleting a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.6 Deleting a VDisk-to-host mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.7 Expanding a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.8 Mapping a VDisk to a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.9 Modifying a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.10 Creating a VDisk Mirror from an existing VDisk . . . . . . . . . . . . . . . . . . . . . . .
413 414 415 418 419 420 421 423 423 424 430 430 431 433 433 434 434 435 436 436 437 438 440 444 450 450 451 453 454 456 457 458 459 460 460 461 463 467 468 470 472 473 474 474 475 477 484 489 490 491 492 493 499
Contents
ix
x
10.8.11 Migrating to a space-efficient VDisk using VDisk mirroring. . . . . . . . . . . . . . . 10.8.12 Splitting a VDisk Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.13 Shrinking a VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.14 Showing the MDisks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.15 Showing the MDisk group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.16 Showing the host to which the VDisk is mapped . . . . . . . . . . . . . . . . . . . . . . 10.8.17 Showing capacity information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.18 Showing VDisks mapped to a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.19 Deleting VDisks from a host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.9 Managing Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.10 Service and maintenance using the GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11 Upgrading software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11.1 Package numbering and version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11.2 Upgrade status utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11.3 Precautions before upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11.4 SVC software upgrade test utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11.5 Running maintenance procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11.6 Setting up error notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11.7 Analyzing the error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11.8 License settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11.9 Viewing the license settings log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.11.10 Listing dumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.12 Backing up the SVC configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.12.1 Backup procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.12.2 Restoring the SVC configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.12.3 Deleting the configuration backup files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
501 504 505 506 507 507 508 509 509 510 510 511 511 511 511 512 519 522 524 528 530 531 535 536 537 537
Chapter 11. Copy Services: FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 SVC FlashCopy features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 How it works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 Implementation of SVC FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.1 FlashCopy mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.2 Multiple Target FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.3 Consistency groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.4 FlashCopy indirection layer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.5 Interaction and dependency between MTFC . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.6 Summary of the FlashCopy indirection layer algorithm. . . . . . . . . . . . . . . . . . . 11.4.7 Interaction with the cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.8 FlashCopy rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.9 FlashCopy and image mode disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.10 FlashCopy mapping events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.11 FlashCopy mapping states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.12 Space-efficient FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.13 Background copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.14 Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.15 Serialization of I/O by FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.16 Error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.17 Asynchronous notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.18 Interoperation with Metro Mirror and Global Mirror . . . . . . . . . . . . . . . . . . . . . 11.4.19 Recovering data from FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 Using the command line to perform FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.1 Scenario description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
539 540 541 542 543 543 544 544 547 549 550 551 551 552 553 555 558 559 559 560 560 562 562 562 563 563
Implementing the IBM System Storage SAN Volume Controller V4.3
11.5.2 Creating a FlashCopy consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.3 Creating a FlashCopy mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.4 Preparing (pre-triggering) the FlashCopy mapping. . . . . . . . . . . . . . . . . . . . . . 11.5.5 Preparing (pre-triggering) the FlashCopy consistency group . . . . . . . . . . . . . . 11.5.6 Starting (triggering) FlashCopy mappings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.7 Starting (triggering) FlashCopy consistency group . . . . . . . . . . . . . . . . . . . . . . 11.5.8 Monitoring the FlashCopy progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.9 Stopping the FlashCopy mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.10 Stopping the FlashCopy consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.11 Deleting the FlashCopy mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.12 Deleting the FlashCopy consistency group. . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.13 Migrate a VDisk to a space-efficient VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6 Using the GUI to perform FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.1 Creating a FlashCopy consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.2 Creating a FlashCopy mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.3 Preparing (pre-triggering) the FlashCopy mapping. . . . . . . . . . . . . . . . . . . . . . 11.6.4 Preparing (pre-triggering) the FlashCopy consistency group . . . . . . . . . . . . . . 11.6.5 Starting (triggering) FlashCopy mappings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.6 Starting (triggering) a FlashCopy consistency group . . . . . . . . . . . . . . . . . . . . 11.6.7 Monitoring the FlashCopy progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.8 Stopping the FlashCopy consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.9 Deleting the FlashCopy mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.10 Deleting the FlashCopy consistency group. . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.11 Migration from a fully allocated VDisk to SEV and vice versa using a GUI. . .
564 565 567 567 568 569 569 570 571 571 572 572 577 577 579 588 589 590 591 592 593 595 595 597
Chapter 12. Copy Services: Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1 Metro Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.1 Metro Mirror overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.2 Remote copy techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.3 SVC Metro Mirror features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.4 Metro Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.5 How Metro Mirror works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.6 Metro Mirror process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.7 Methods of synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.8 State overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.9 Detailed states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.10 Practical use of Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.11 Metro Mirror configuration limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Metro Mirror commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.1 Listing available SVC cluster partners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.2 Creating SVC cluster partnership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.3 Creating a Metro Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.4 Creating a Metro Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.5 Changing a Metro Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.6 Changing a Metro Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.7 Starting a Metro Mirror relationship. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.8 Stopping a Metro Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.9 Starting a Metro Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.10 Stopping a Metro Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.11 Deleting a Metro Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.12 Deleting a Metro Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.13 Reversing a Metro Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.14 Reversing a Metro Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . .
603 604 604 604 606 607 611 612 612 615 618 621 622 622 623 623 624 624 625 625 625 626 626 627 627 627 628 628
Contents
xi
xii
12.2.15 Background copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Metro Mirror scenario using the CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.1 Setting up Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.2 Starting Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.3 Stopping and restarting Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.4 Changing copy direction for Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4 Metro Mirror scenario using the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.1 Setting up Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.2 Starting Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.3 Stopping and restarting Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.4 Changing copy direction for Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
628 629 630 634 637 640 642 643 657 660 666
Chapter 13. Copy Services: Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1 Global Mirror overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.1 Intracluster Global Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.2 Intercluster Global Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2 Remote copy techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.1 Asynchronous remote copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.2 Synchronous remote copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.3 SVC Global Mirror features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.4 Global Mirror relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.5 How Global Mirror works. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.6 Global Mirror process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.7 Methods of synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.8 State overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.9 Detailed states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.10 Practical use of Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.11 Global Mirror configuration limits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3 Global Mirror commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.1 Listing the available SVC cluster partners . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.2 Creating an SVC cluster partnership. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.3 Creating a Global Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.4 Creating a Global Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.5 Changing a Global Mirror relationship. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.6 Changing a Global Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.7 Starting a Global Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.8 Stopping a Global Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.9 Starting a Global Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.10 Stopping a Global Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.11 Deleting a Global Mirror relationship. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.12 Deleting a Global Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.13 Reversing a Global Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.14 Reversing a Global Mirror consistency group . . . . . . . . . . . . . . . . . . . . . . . . . 13.4 Global Mirror scenario using the CLI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.1 Setting up Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.2 Starting Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.3 Stopping and restarting Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.4 Changing direction for Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5 Global Mirror scenario using the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.1 Setting up Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.2 Starting Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.3 Stopping and restarting Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.4 Changing copy direction for Global Mirror. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
669 670 670 670 670 670 671 672 673 677 678 678 681 684 687 688 689 689 690 691 691 692 692 692 693 693 694 694 694 695 695 695 696 702 704 708 710 711 728 731 737
Implementing the IBM System Storage SAN Volume Controller V4.3
Chapter 14. Migration to and from the SAN Volume Controller . . . . . . . . . . . . . . . . . 14.1 Migration overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2 Migration operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2.1 Migrating multiple extents (within an MDG) . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2.2 Migrating extents off an MDisk that is being deleted. . . . . . . . . . . . . . . . . . . . . 14.2.3 Migrating a VDisk between MDGs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2.4 Migrating the VDisk to image mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2.5 Migrating a VDisk between I/O groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2.6 Monitoring the migration progress. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3 Functional overview of migration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.1 Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.2 Error handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.3 Migration algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4 Migrating data from an image mode VDisk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.1 Image mode VDisk migration concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.2 Migration tips. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5 Data migration for Windows using the SVC GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5.1 SVC added between the host system and the DS4700 . . . . . . . . . . . . . . . . . . 14.5.2 Put the migrated disks on a Windows 2008 host online . . . . . . . . . . . . . . . . . . 14.5.3 Migrating the VDisk from image mode to managed mode . . . . . . . . . . . . . . . . 14.5.4 Migrating the VDisk from managed mode to image mode . . . . . . . . . . . . . . . . 14.5.5 Migrating the VDisk from image mode to image mode . . . . . . . . . . . . . . . . . . . 14.5.6 Free the data from the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5.7 Put the disks online in Windows 2008 that have been freed from SVC . . . . . . 14.6 Migrating Linux SAN disks to SVC disks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6.1 Connecting the SVC to your SAN fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6.2 Prepare your SVC to virtualize disks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6.3 Move the LUNs to the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6.4 Migrate the image mode VDisks to managed MDisks . . . . . . . . . . . . . . . . . . . 14.6.5 Preparing to migrate from the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6.6 Migrate the VDisks to image mode VDisks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6.7 Remove the LUNs from the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7 Migrating ESX SAN disks to SVC disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7.1 Connecting the SVC to your SAN fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7.2 Prepare your SVC to virtualize disks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7.3 Move the LUNs to the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7.4 Migrate the image mode VDisks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7.5 Preparing to migrate from the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7.6 Migrate the managed VDisks to image mode VDisks . . . . . . . . . . . . . . . . . . . . 14.7.7 Remove the LUNs from the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.8 Migrating AIX SAN disks to SVC disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.8.1 Connecting the SVC to your SAN fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.8.2 Prepare your SVC to virtualize disks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.8.3 Move the LUNs to the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.8.4 Migrate image mode VDisks to VDisks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.8.5 Preparing to migrate from the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.8.6 Migrate the managed VDisks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.8.7 Remove the LUNs from the SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.9 Using SVC for storage migration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
741 742 742 742 743 744 746 746 748 749 749 750 750 751 751 754 754 758 765 768 771 775 779 781 783 785 786 790 793 796 799 800 803 804 806 810 813 816 818 819 822 824 825 830 832 834 837 838 841
Appendix A. Copy Services and open systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 843 AIX specifics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 844 AIX and FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 844
Contents
xiii
AIX and Metro Mirror and Global Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Making updates to the LVM information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Windows NT and 2000/2003 specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Windows NT and Copy Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Copy Services with Windows NT Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Windows 2000/2003 and Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
858 859 860 860 860 865
Appendix B. DS4000 and DS8000 migration scenarios. . . . . . . . . . . . . . . . . . . . . . . . Initial considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Device recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Scenario 1: DS4000 Total number of LUNs is less than maximum LUNs per partition . . . Scenario 2: DS4000 total number of LUNs is more than the maximum LUNs per partition Scenario 3: Migrating DS8000 Storage to SVC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
885 886 887 888 892 896
Appendix C. Scripting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Scripting structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Automated VDisk creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SVC tree. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Scripting alternatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
901 902 903 906 913
Appendix D. Node replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Replacing nodes nondisruptively . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Expanding an existing SVC cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Moving VDisks to a new I/O group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Replacing nodes disruptively (rezoning the SAN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
915 916 920 922 923
Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Other resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Referenced Web sites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How to get Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
925 925 925 926 926 927
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 929
xiv
Implementing the IBM System Storage SAN Volume Controller V4.3
Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.
© Copyright IBM Corp. 2003-2008. All rights reserved.
xv
Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: AIX 5L™ AIX® BladeCenter® Chipkill™ DB2® developerWorks® DS4000™ DS6000™ DS8000™
Enterprise Storage Server® FlashCopy® GPFS™ IBM® Power Systems™ Redbooks® Redbooks (logo) ® RS/6000® System i®
System p® System Storage™ System Storage DS® System x™ System z® Tivoli® TotalStorage® WebSphere®
The following terms are trademarks of other companies: Disk Magic, and the IntelliMagic logo are trademarks of IntelliMagic BV in the United States, other countries, or both. Data ONTAP, NetApp, and the NetApp logo are trademarks or registered trademarks of NetApp, Inc. in the U.S. and other countries. Novell, SUSE, the Novell logo, and the N logo are registered trademarks of Novell, Inc. in the United States and other countries. Oracle, JD Edwards, PeopleSoft, Siebel, and TopLink are registered trademarks of Oracle Corporation and/or its affiliates. QLogic, and the QLogic logo are registered trademarks of QLogic Corporation. SANblade is a registered trademark in the United States. VMotion, VMware, the VMware "boxes" logo and design are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. JNI, Solaris, Sun, and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Active Directory, Internet Explorer, Microsoft, MS-DOS, Windows NT, Windows Server, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel Xeon, Intel, Itanium-based, Itanium, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.
xvi
Implementing the IBM System Storage SAN Volume Controller V4.3
Preface This IBM® Redbooks® publication is a detailed technical guide to the IBM System Storage™ SAN Volume Controller (SVC), a virtualization appliance solution that maps virtualized volumes visible to hosts and applications to physical volumes on storage devices. Each server within the SAN has its own set of virtual storage addresses, which are mapped to physical addresses. If the physical addresses change, the server continues running using the same virtual addresses that it had before. This means that volumes or storage can be added or moved while the server is still running. The IBM virtualization technology improves management of information at the “block” level in a network, enabling applications and servers to share storage devices on a network. Successful businesses require real-time responsiveness to change, whether because of new customer needs, changes in the supply chain, unexpected competitive moves, external threats, or changes in the economic climate. Rapid response to change requires an IT infrastructure that can turn information into a competitive advantage; the IT infrastructure must provide the maximum benefit at an affordable cost, and must have the flexibility to support changes in business processes. An on demand operating environment provides a cost effective and flexible IT environment. With information at the heart of competitiveness, storage becomes an ever more critical component of an on demand operating environment. The IBM System Storage strategy addresses some of the most pressing needs currently facing Chief Information Officers (CIO) and IT managers. As part of its strategy, IBM intends to deliver industry leading technologies that will help dramatically reduce the total cost of ownership (TCO) for storage, and help turn fixed costs into variable costs that scale with business volume. Success in the on demand world will depend on the ability to leverage information technology. A greater dependence on information means a greater dependence on storage. What distinguishes an on demand business is the ability to quickly sense and rapidly respond to a dynamic marketplace; to do this, there are challenges that an on demand business must overcome. At the business level, customers are faced with three major storage challenges: Managing storage growth: Storage needs to continue to grow at over 50% per year. Managing storage is becoming more complex than ever, because we now have to deal with multiple server platforms and different operating systems, which may be connected to a storage area network (SAN) with multiple and diverse storage platforms. Increasing complexity: Although the declining cost of storage per megabyte makes it attractive to add additional disks, the increasing complexity of managing this storage results in overutilized staff and underutilized IT resources. Combining this with the shortage of skilled storage administrators, it is possible to add significant cost and introduce risk to storage management. Maintaining availability: The added complexity of 24x7 environments significantly reduces, for example, the efficiency of conducting routine maintenance, scheduling backups, data migration, and introducing new software and hardware. This problem is compounded by the fact that as availability increases, so does the cost inherent with making it so. Variety of information: Information technology holds the promise of bringing a variety of new types of information to the people who need it. Volume of data: Data is growing exponentially. Estimates show a continued 60% yearly growth of new disk in petabytes shipped. © Copyright IBM Corp. 2003-2008. All rights reserved.
xvii
Velocity of change: IT organizations are under tremendous pressure to deliver the right IT services. Approximately 85% of problems are caused by IT staff changing something, and 80% of problems are not detected by IT staff until reported. These challenges still exist, although large SANs do offer desirable and tangible benefits, for example, better connectivity, improved performance, distance flexibility, and scalability. However, even these benefits may be outweighed by the added complexity that they introduce. As an example, large enterprise SANs often contain different types of storage devices. These differences could be in the types of disk deployed, their level of performance, or the functionality provided, such as RAID or mirroring. Often, customers have different vendor storage devices as the result of mergers or consolidations. The result, however, is that storage and SAN administrators need to configure storage to servers, and then keep track of which servers own or have access to that storage. The storage administrative tasks can become daunting as the SAN grows and as the storage administrators manually attempt to manage the SAN. Furthermore, the complexity of having different file systems in the same SAN requires that storage administrators know how to administer each client operating system (OS) platform. The management interfaces for each may be different, since there is no common standard that all vendors adhere to. Lastly, since the file systems are tied to each of the servers, storage management functions potentially have to be run on hundreds of servers. It is easy to see why manageability and interoperability are the top areas for concern, especially in a SAN where the number of possible storage and OS platform permutations is considerable. These challenges are at odds with the commonly held belief that storage is decreasing in cost per megabyte. It is clear that the cost of managing storage is greater than the initial purchase price. A product is needed to address storage manageability, while at the same time addressing the need for interoperability. This product is the focus of this book.
The team that wrote this book This book was produced by a team of specialists from around the world working at the International Technical Support Organization, San Jose Center. Jon Tate is a Project Manager for IBM System Storage SAN Solutions at the International Technical Support Organization, San Jose Center. Before joining the ITSO in 1999, he worked in the IBM Technical Support Center, providing Level 2 support for IBM storage products. Jon has 23 years of experience in storage software and management, services, and support, and is both an IBM Certified IT Specialist and an IBM SAN Certified Specialist. Jon also serves as the UK Chair of the Storage Networking Industry Association. Sameer Dhulekar is a Senior IT Specialist for Systems Solutions Center, at the System and Technology Group, India. Sameer has 13 years of experience in Storage, Storage Area Networks, UNIX®, database and software management, services, and support. Sameer specializes in benchmark, performance, virtualization, and high availability solutions. He has participated in multiple storage seminars and forums. Juerg Hossli is an Advisory Technical Services Professional at Global Technology Services, Integrated Technology Delivery in Switzerland. He has 15 years of experience in the IT field and has worked at IBM for eleven years. He holds a Swiss Federal Certificate in Computer Science. His areas of expertise include planning, implementation, and maintenance of high-end storage solutions, storage virtualization, the SAN environment, and Tivoli® Storage products. xviii
Implementing the IBM System Storage SAN Volume Controller V4.3
Dan Koeck is a Storage and IBM System x™ Specialist in IBM Maintenance and Technical Support (MTS) in Austria. He has a graduate degree in applied computer science and has many industry certifications. In the very near future, he will join the IBM STG pre-sales group in Austria and is working towards a second degree in IT Security. He has worked at IBM for six years, and his areas of expertise include disk storage, SAN, IBM System x, BladeCenter®, server and storage virtualization, high-availability solutions, data center migration, and software engineering. He previously authored Tuning IBM System x Servers for Performance, SG24-5287 and IBM System Storage DS3000: Introduction and Implementation Guide, SG24-7065. Suad Musovich is a Senior IT Specialist for IBM Global Technology Services in New Zealand. He has seven years of experience working with IBM Storage Systems, and in his current role he is involved with the planning, design, implementation, management, and problem analysis of IBM storage solutions. His areas of expertise is the SAN infrastructure, disk storage, UNIX systems, and IBM Tivoli Storage Manager backup solutions.
Figure 1 Left to right: Dan, Juerg, Sam, Suad, and Jon
We extend our thanks to the following people for their contributions to this project. There are many people that contributed to this book. In particular, we thank the development and PFE teams in Hursley. Matt Smith was also instrumental in moving any issues along and ensuring that they maintained a high profile.
Preface
xix
In particular, we thank the authors of the previous editions of this book: Matt Amanat Angelo Bernasconi Steve Cody Sean Crawford Katja Gebuhr Deon George Amarnath Hiriyannappa Thorsten Hoss Philippe Jachimczyk Kamalakkannan J Jayaraman Bent Lerager Craig McKenna Andy McManus Joao Marcos Leite Barry Mellish Massimo Rosati Fred Scholten Robert Symons Marcus Thordal Xiao Peng Zhao We would also like to thank the following people for their contributions: John Agombar Alex Ainscow Trevor Boardman Chris Canto Peter Eccles Carlos Fuente Alex Howell Colin Jewell Paul Mason Paul Merrison Jon Parkes Steve Randle Lucy Raw Bill Scales Dave Sinclair Matt Smith Steve White Barry Whyte IBM Hursley Bill Wiegand IBM Advanced Technical Support Timothy Crawford Ross Hagglund IBM Beaverton Dorothy Faurot IBM Raleigh Marci Nagel IBM Rochester
xx
Implementing the IBM System Storage SAN Volume Controller V4.3
Chris Saul IBM San Jose Glen Routley IBM Australia Sharon Wang IBM Chicago Deanna Polm Sangam Racherla IBM ITSO
Become a published author Join us for a two- to six-week residency program! Help write a book dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You will have the opportunity to team with IBM technical professionals, Business Partners, and Clients. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you will develop a network of contacts in IBM development labs, and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html
Comments welcome Your comments are important to us! We want our books to be as helpful as possible. Send us your comments about this book or other IBM Redbooks in one of the following ways: Use the online Contact us review Redbooks form found at: ibm.com/redbooks Send your comments in an e-mail to:
[email protected] Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HYTD Mail Station P099 2455 South Road Poughkeepsie, NY 12601-5400
Preface
xxi
xxii
Implementing the IBM System Storage SAN Volume Controller V4.3
Summary of changes This section describes the technical changes made in this edition of the book and in previous editions. This edition may also include minor corrections and editorial changes that are not identified. Summary of Changes for SG24-6423-06 for Implementing the IBM System Storage SAN Volume Controller V4.3 as created or updated on October 24, 2008.
October 2008, Seventh Edition This revision reflects the addition, deletion, or modification of new and changed information described below.
New information
SVC Console Space-efficient VDisks VDisk Mirroring IPv6 Additional host support
© Copyright IBM Corp. 2003-2008. All rights reserved.
xxiii
xxiv
Implementing the IBM System Storage SAN Volume Controller V4.3
1
Chapter 1.
Introduction to storage virtualization In this chapter, we describe the need for storage virtualization and the IBM approach to both in-band and out-of-band storage virtualization. The fundamental differences between the two architectures are articulated to explain why IBM has chosen to use in-band virtualization for the IBM System Storage SAN Volume Controller (the focus of the remainder of this book).
© Copyright IBM Corp. 2003-2008. All rights reserved.
1
1.1 The need for storage virtualization At the business level, clients are faced with three major storage challenges: Managing storage growth: Storage needs continue to grow at a rate that is normally higher than what has been planned for each year. As an example, storage subsystems can be purchased to last for 3 to 5 years; however, organizations are finding that they are filling to capacity much earlier than that. To fill the growth, customers are then either extending their current storage subsystems in chunks, or buying different types of storage subsystems to match their storage needs and budget. Increasing complexity: As storage needs grow, this need can be filled by more than one disk subsystem, which might not even be from the same vendor. Together with the variety of server platforms and operating systems in a customer’s environment, customers can have storage area networks (SAN) with multiple and diverse storage subsystems and host platforms. Combining this with the shortage of skilled storage administrators, the cost and risk of storage increases as the environment becomes more complex. Maintaining availability: With the increased range of storage options available, the storage growth rate, and no similar increase in storage budget, customers are having to manage more storage with minimal or no additional staff. Thus, with the complexity highlighted above, and with business requirements on IT resources demanding higher business system availability, the room for errors increases as each new storage subsystem is added to the infrastructure. Additionally, making changes to the storage infrastructure to accommodate storage growth traditionally leads to outages that might not be acceptable by the business. Storage needs are rising, and the challenge of managing disparate storage systems is growing. The IBM System Storage SAN Volume Controller brings storage devices together in a virtual pool to make all storage appear as: One “logical” device to centrally manage and to allocate capacity as needed One solution to help achieve the most effective use of key storage resources on demand Virtualization solutions can be implemented in the storage network, in the server, or in the storage device itself. The IBM storage virtualization solution is SAN-based, which helps allow for a more open virtualization implementation. Locating virtualization in the SAN, and therefore in the path of input/output (I/O) activity, helps provide a solid basis for policy-based management. The focus of IBM on open standards means its virtualization solution supports freedom of choice in storage-device vendor selection. The IBM System Storage SAN Volume Controller solution is designed to: Simplify storage management Reduce IT data storage complexity and costs while enhancing scalability Extend on demand flexibility and resiliency to the IT infrastructure Increase application availability by making changes in the infrastructure without having to shut down hosts
2
Implementing the IBM System Storage SAN Volume Controller V4.3
1.2 In-band virtualization In a conventional SAN, the logical unit numbers (LUNs) that are defined within the storage subsystem are directly presented to the host or hosts. In-band virtualization, otherwise known as block aggregation, essentially means having an appliance in the data path that can take physical storage from one or more storage subsystems and offer it to hosts in the form of a virtual disk (VDisk). The Storage Networking Industry Association (SNIA) block aggregation model (Figure 1-1) specifies that block aggregation can be performed within hosts (servers), in the storage network (storage routers and storage controllers), or in storage devices (intelligent disk arrays).
Application
Storage devices (disks, …)
Capacity planning
Device-based block aggregation
High availability (fail-over, …)
SN-based block aggregation
Redundancy mgmt (backup, …)
Block aggregation
Security, billing
Host-based block aggregation
Discovery, monitoring
File system (FS)
Services subsystem
Storage domain
Database (dbms)
Resource mgmt, configuration
File/record subsystem
Block subsystem Copyright 2000, Storage Network Industry Association
Figure 1-1 SNIA block aggregation model
While each of these approaches has pros and cons and all are available in various forms from various vendors, IBM chose to develop its latest block aggregation product (IBM System Storage SAN Volume Controller) within the storage network. Block aggregation within the storage network provides four significant benefits to clients: Increased storage administrator productivity: Administrators can manage, add, and migrate physical disks non-disruptively from an application server point of view. This is accomplished by providing insulation between the server’s view of the logical disks and the disks as presented by the storage subsystem. Productivity is improved by allowing administrators to perform management functions when convenient rather than waiting for ever decreasing maintenance windows. Downtime requirements are almost eliminated.
Chapter 1. Introduction to storage virtualization
3
Providing a common platform for advanced functions: By providing a logical view of physical storage, advanced functions like disaster recovery can be done at a single point in the SAN in a consistent way regardless of the underlying physical storage. FlashCopy®, Metro Mirror (formerly referred to as Peer-to-Peer Remote Copy (PPRC)) and data migration can also be performed in a consistent way. This common platform is used to provide other advanced functions over time, such as advanced security and quality of service (QoS) capabilities. Improved capacity utilization: Spare capacity on underlying physical disks can be reallocated non-disruptively from an application server point of view irrespective of the server operating system or platform type. Logical disks can be created from any of the physical disks being managed by the virtualization device (that is, vendor agnostic). Simplification of connectivity: Each vendor storage subsystem would traditionally require a vendor’s device driver on the host to access the subsystem. Where there are many subsystems in the environment, regardless of whether any one host is accessing more than one vendor’s storage subsystems, then managing the range of device drivers is unnecessarily complex. The IBM approach means that only one device driver, the IBM System Storage Subsystem Device Driver (SDD), is required to access any virtualized storage on the SAN regardless of the vendor storage subsystem. Figure 1-2 shows the IBM approach to block aggregation.
Figure 1-2 IBM plan for block aggregation
4
Implementing the IBM System Storage SAN Volume Controller V4.3
In addition to the four major benefits outlined above, abstracting the hosts from directly accessing the storage subsystem or subsystems has many other benefits over other methods of block aggregation, including these: It provides the ability to add advanced functions and apply them to the entire storage infrastructure. The first release of the product offered these functions: – Copy Services (Metro Mirror (formerly referred to as PPRC) and FlashCopy) – Data migration – Read and Write Caching Later releases of the product offer such functions as: – Quality of Service – Performance based data migration – Performance optimization in the data path – Advanced security – Copy Services: Global Mirror – Space-efficiency
It does not lock a client into a particular storage hardware vendor. It is not intrusive on the hosts. It can offload functionality from the hosts. It can support storage management from multiple ISVs. It offers superior scalability.
The IBM virtualization product provides redundant, modular, and scalable solutions. It is based on a clustered IBM SAN appliance running a Linux® kernel to support high availability and performance. Additional nodes are capable of being added non-disruptively, providing enterprise class scalability. IBM’s long history of storage controller development has enabled us to develop systems where, in the exceptionally rare case that a failure occurs, the virtualization device can fail and recover gracefully.
Chapter 1. Introduction to storage virtualization
5
Figure 1-3 shows a representation of the IBM System Storage SAN Volume Controller.
Figure 1-3 Conceptual diagram of the IBM SAN Volume Controller
In summary, enterprise class block aggregation functionality is added to the storage network. The IBM solution improves storage administrator productivity, provides a common base for advanced functions, and provides for more efficient use of storage. The IBM product is designed to be delivered as a horizontally scalable, integrated solution based on the IBM SAN appliance, and Linux, using a fault tolerant clustered architecture.
1.3 Out-of-band virtualization Out-of-band virtualization, otherwise known as file aggregation, is when the virtualization appliance is not in the data path. Typically, out-of-band virtualization is more geared towards file sharing across the SAN. To this end, it typically involves a single file system in a single name space. File aggregation is a similar technique to block aggregation. However, rather than dealing with blocks of data, file aggregation addresses the needs of accessing and sharing files in a storage network. In the SNIA model, hosts get file metadata from file system or Network Attached Storage (NAS) controllers, and then access the data directly. File aggregation can be used in conjunction with or independent from block aggregation. Figure 1-4 on page 7 shows the SNIA file aggregation model.
6
Implementing the IBM System Storage SAN Volume Controller V4.3
Figure 1-4 SNIA file aggregation model
1.3.1 IBM Global Parallel File System The IBM approach to file virtualization is through the use of a common file system based on the IBM Global Parallel File System (GPFS™). GPFS is a high-performance shared-disk cluster file system. GPFS provides concurrent high-speed file access to applications executing on multiple nodes of an AIX® 5L™ cluster, a Linux cluster, or a heterogeneous cluster of AIX 5L and Linux nodes. In addition to providing file system storage capabilities, GPFS provides tools for management and administration of the GPFS cluster and allows for shared access to file systems from remote GPFS clusters. Note: Effective April 20, 2007, IBM withdrew from marketing the SAN File System. Its replacement is GPFS. GPFS provides scalable, high-performance data access from a single node to 2,000 nodes or more. Up to 512 Linux nodes or 128 AIX 5L nodes with access to one or more file systems are supported as a general statement and larger configurations exist by special arrangements with IBM. A GPFS file system is built from a collection of disks that contain the file system data and metadata. A file system can be built from a single disk or contain thousands of disks, each up to 2 terabytes in size, storing petabytes of data. GPFS provides file level virtualization through a rule based policy engine. Data from files in a in a single directory can reside in one or across several storage pools. Determination of which storage pool the file data is initially written to, and how it is migrated, mirrored, or deleted over its life span, is based on a set of business rules in an administrator defined policy.
Chapter 1. Introduction to storage virtualization
7
GPFS enables file level virtualization. This allows clients to reap the benefit of better application business responsiveness, maximized storage utilization, dynamic resource allocation, improved storage administration utilization, and reduced storage outage. For more details about the technology and implementation of GPFS, see the white paper, An Introduction to GPFS, at: http://www-03.ibm.com/systems/clusters/software/whitepapers/gpfs_intro.html
1.4 Conclusion In conclusion, the IBM System Storage SAN Volume Controller enables storage virtualization. This allows clients to reap the benefit of better application business responsiveness, maximized storage utilization, dynamic resource allocation, improved storage administration utilization, and reduced storage outage. In-band and out-of-band virtualization provide two very distinct yet complementary approaches to virtualization. IBM will extol the virtues of each in two separate products. Both products fulfill different requirements, and therefore use different approaches to virtualization. The rest of this book is dedicated to the IBM System Storage SAN Volume Controller and its method of in-band virtualization.
8
Implementing the IBM System Storage SAN Volume Controller V4.3
2
Chapter 2.
IBM System Storage SAN Volume Controller overview In this chapter, we describe the major concepts behind the IBM System Storage SAN Volume Controller to provide the framework for the discussion of the remainder of this book.
© Copyright IBM Corp. 2003-2008. All rights reserved.
9
2.1 Maximum supported configurations For a list of the maximum supported configurations, visit the SVC support site at: http://www.ibm.com/storage/support/2145
2.2 Glossary of commonly used terms Before providing an overview of the IBM System Storage SAN Volume Controller, we begin this chapter with a short glossary of terms (in alphabetical order) most commonly used throughout this book.
Configuration node While the cluster is operational, a single node in the cluster is appointed to provide configuration and service functions over the network interface. This node is termed the configuration node. This configuration node manages a cache of the configuration information that describes the cluster configuration and provides a focal point for configuration commands. If the configuration node fails, another node in the cluster will assume the role.
Extent To track the space that is available on an MDisk, the SAN Volume Controller divides each MDisk into chunks of equal size. These chunks are called extents and are indexed internally
Front end and back end The SAN Volume Controller takes managed disks and presents these to application servers (hosts). The managed disks are looked after by the “back-end” application of the SAN Volume Controller. The virtual disks presented to hosts are looked after by the “front-end” application in the SAN Volume Controller.
Grain A grain is the unit of data represented by a single bit in a FlashCopy bitmap (64 KB / 256 KB) in the SAN Volume Controller. It is also the unit to extend the real size of a space-efficient VDisk (32,64,128 or 256 KB).
I/O group An input/output (I/O) group contains two SAN Volume Controller nodes defined by the configuration process. Each SAN Volume Controller node is associated with exactly one I/O group. The nodes in the I/O group provide access to the VDisks in the I/O group.
LU and LUN Strictly speaking, there is a difference between a logical unit (LU) and a logical unit number (LUN). A LUN is a unique identifier used on a SCSI bus that enables it to differentiate between devices (each of which is a logical unit). Each of the LUs in the SVC is an MDisk. In practice, the two terms are used interchangeably. In this book, when we refer to a LUN, we refer to the unit of storage that is defined in a storage subsystem such as an IBM System Storage Enterprise Storage Server® (ESS), IBM System Storage DS3000,DS4000™, DS6000™, and DS8000™ series Storage Server, or storage servers from other vendors.
Managed disk A managed disk (MDisk) is a logical disk (typically a RAID or partition thereof) that a storage subsystem has exported to the SAN fabric to which the nodes in the cluster are attached.
10
Implementing the IBM System Storage SAN Volume Controller V4.3
Managed disk group A managed disk (MDisk) group is a collection of MDisks.
Master console The master console is the platform on which the software used to manage the SAN Volume Controller runs. With Version 4.3. it is being replaced by SSPC. However, V4.3 GUI Console code is supported on existing master consoles.
Node A SAN Volume Controller node is a single processing unit, which provides virtualization, cache, and copy services for the SAN. Nodes are deployed in pairs called I/O groups. One node in the cluster is designated the configuration node, but each node in the cluster holds a copy of the cluster state information.
SAN Volume Controller The SAN Volume Controller is a SAN appliance designed for attachment to a variety of host computer systems, which carries out block level virtualization of disk storage.
SSPC IBM System Storage Productivity Center (SSPC) replaces the master console for new installations of SAN Volume Controller Version 4.3.0. For SSPC planning, installation, and configuration information, see the following Web site: http://publib.boulder.ibm.com/infocenter/tivihelp/v4r1/index.jsp
Virtual disk A virtual disk (VDisk) is a SAN Volume Controller device that appears to host systems attached to the SAN as a SCSI disk. Each VDisk is associated with exactly one I/O group.
2.3 Virtualization overview The SVC nodes are the hardware elements of the IBM System Storage SAN Volume Controller, a member of the IBM System Storage virtualization family of solutions. The SAN Volume Controller combines servers into a high availability cluster. Each of the servers in the cluster is populated with 8 GB of high-speed memory, which serves as the cluster cache. A management card is installed in each server to monitor various parameters that the cluster uses to determine the optimum and continuous data path. The cluster is protected against data loss by uninterruptible power supplies. The SAN Volume Controller nodes can only be installed in pairs to avoid a single point of failure. Storage virtualization addresses the increasing cost and complexity in data storage management. It addresses this increased complexity by shifting storage management intelligence from individual SAN disk subsystem controllers into the network through a virtualization cluster of nodes. The SAN Volume Controller solution is designed to reduce both the complexity and costs of managing your SAN-based storage. With the SAN Volume Controller, you can: Simplify management and increase administrator productivity by consolidating storage management intelligence from disparate disk subsystem controllers into a single view. Improve application availability by enabling data migration between disparate disk storage devices non-disruptively.
Chapter 2. IBM System Storage SAN Volume Controller overview
11
Improve disaster recovery and business continuance needs by applying and managing copy services across disparate disk storage devices within the Storage Area Network (SAN). These solutions include a Common Information Model (CIM) Agent, enabling unified storage management based on open standards for units that comply with CIM Agent standards. Provide advanced features and functions to the entire SAN, such as: – Large scalable cache – Copy Services – Space management (later releases to include Policy Based Management) – Mapping based on desired performance characteristics – Quality of Service (QoS) metering and reporting Simplify device driver configuration on hosts, so all hosts within your network use the same IBM device driver to access all storage subsystems through the SAN Volume Controller. Note: The SAN Volume Controller is not a RAID controller. The disk subsystems attached to SANs that have the SAN Volume Controller provide the basic RAID setup. The SAN Volume Controller uses what is presented to it as a managed disk to create virtual disks.
2.4 Compass architecture The IBM System Storage SAN Volume Controller is based on the Commodity Parts Storage System (Compass) architecture developed at the IBM Almaden Research Center. The overall goal of the Compass architecture is to create storage subsystem software applications that require minimal porting effort to leverage a new hardware platform. To meet this goal: Compass, although currently deployed on the Intel® hardware platform, can be ported to other hardware platforms. Compass, although currently deployed on a Linux kernel, can be ported to other Portable Operating System Interface (POSIX)-compliant operating systems. Compass uses commodity adapters and parts wherever possible. To the highest extent possible, it only uses functions in the commodity hardware that are commonly exercised by the other users of the parts. This is not to say that Compass software could not be ported to a platform with specialized adapters. However, the advantage in specialized function must be weighed against the disadvantage of future difficulty in porting and in linking special hardware development plans to the release plans for applications based on the Compass architecture. Compass is developed in such a way that it is as easy as possible to troubleshoot and correct software defects. Compass is designed as a scalable, distributed software application that can run in increasing sets of Compass nodes with near linear gain in performance while using a shared data model that provides a single pool of storage for all nodes. Compass is designed so that there is a single configuration and management view of the entire environment regardless of the number of Compass nodes in use.
12
Implementing the IBM System Storage SAN Volume Controller V4.3
The approach is to minimize the dependency on unique hardware, and to allow exploitation of or migration to new SAN interfaces simply by plugging in new commodity adapters. Performance growth over time is ensured by the ability to port Compass to just about any platform and remain current with the latest processor and chipset technologies on each. The SAN Volume Controller implementation of the Compass architecture has exploited Linux as a convenient development platform to deploy this function. This has enhanced and will continue to enhance the ability of IBM to deploy robust function in a timely way. SVC relies on the Compass architecture to provide high levels of fault tolerance and high availability. Extensive dump capabilities are provided to enable first failure capture of software defects. Fault tolerance and high levels of availability are achieved by:
The RAID capabilities of the underlying disk subsystems SVC clustering using the Compass architecture Auto-restart of hung nodes UPS units to provide memory protection in the event of a site power failure Host System Failover capabilities
High levels of serviceability are achieved by providing:
Cluster error logging Asynchronous error notification Dump capabilities to capture software detected failures Concurrent diagnostics Directed maintenance procedures Concurrent log analysis and dump data recovery tools Concurrent maintenance of all SVC components Concurrent upgrade of SVC software and microcode Concurrent addition or deletion of SVC nodes in a cluster Software recovery through a service panel push button Automatic software version correction when replacing a node Detailed status and error conditions displayed on the service panel Error and event notification through SNMP and e-mail
Support is provided for the end-to-end SAN problem determination through detailed fabric status reporting capabilities.
2.4.1 SAN Volume Controller clustering In simple terms, a cluster is a collection of servers that, together, provide a set of resources to a client. The key point is that the client has no knowledge of the underlying physical hardware of the cluster. This means that the client is isolated and protected from changes to the physical hardware, which brings a number of benefits. Perhaps the most important of these benefits is high availability. Resources on clustered servers act as highly available versions of unclustered resources. If a node (an individual computer) in the cluster is unavailable, or too busy to respond to a request for a resource, the request is transparently passed to another node capable of processing it, so clients are unaware of the exact locations of the resources they are using.
Chapter 2. IBM System Storage SAN Volume Controller overview
13
For example, a client can request the use of an application without being concerned about either where the application resides or which physical server is processing the request. The user simply gains access to the application in a timely and reliable manner. Another benefit is scalability: If you need to add users or applications to your system and want performance to be maintained at existing levels, additional systems can be incorporated into the cluster. The IBM System Storage SAN Volume Controller is a collection of up to eight cluster nodes, added in pairs. In future releases, the cluster size will be increased to permit further performance scalability. These nodes are managed as a set (cluster) and present a single point of control to the administrator for configuration and service activity. Note: Although the SAN Volume Controller code is based on a Linux kernel, the clustering feature is not based on Linux clustering code. The clustering failover and failback feature is part of the SAN Volume Controller application software. Within each cluster, one node is defined as the configuration node. This node is assigned the cluster IP address and is responsible for transitioning additional nodes into the cluster. During normal operation of the cluster, the nodes communicate with each other. If a node is idle for a few seconds, then a heartbeat signal is sent to ensure connectivity with the cluster. Should a node fail for any reason, the workload intended for it is taken over by another node until the failed node has been restarted and re-admitted to the cluster (which happens automatically). In the event that the microcode on a node becomes corrupted, resulting in a failure, the workload is transferred to another node. The code on the failed node is repaired, and the node is re-admitted to the cluster (again, all automatically). For I/O purposes, SAN Volume Controller nodes within the cluster are grouped into pairs, called I/O groups, with a single pair being responsible for serving I/O on a given VDisk. One node within the I/O group represents the preferred path for I/O to a given VDisk. The other node represents the non-preferred path. This preference alternates between nodes as each VDisk is created within an I/O group to balance the workload evenly between the two nodes. Note: The preferred node by no means signifies absolute ownership. The data can still be accessed by the partner node in the I/O group in the event of a failure. Beyond automatic configuration and cluster administration, the data transmitted from attached application servers is also treated in the most reliable manner. When data is written by the host, the preferred node within the I/O group stores a write in its own write cache and the write cache of its partner (non-preferred) node before sending an “I/O complete” status back to the host application. The write cache is automatically destaged to disk after two minutes of no writes to a VDisk. To ensure that data is written in the event of a node failure, the surviving node empties all of its remaining write cache and proceeds in write-through mode until the cluster is returned to a fully operational state. Note: Write-through mode is where the data is not cached in the nodes, but written directly to the disk subsystem instead. While operating in this mode, performance can be degraded. More importantly, it ensures that the data makes it to its destination without the risk of data loss. A single copy of data in cache would constitute exposure to data loss.
14
Implementing the IBM System Storage SAN Volume Controller V4.3
Another data protection feature that the SAN Volume Controller has is uninterruptible power supply units. In addition to voltage regulation to protect valuable electronic components within the SAN Volume Controller configuration, in the event of a main power outage, the uninterruptible power supply provides enough power to destage data to the SAN Volume Controller internal disk and shut down the nodes within the SAN Volume Controller cluster gracefully. This is a feature found in most high-end disk subsystems.
2.4.2 SAN Volume Controller virtualization The SAN Volume Controller provides block aggregation and volume management for disk storage within the SAN. In simpler terms, this means that the SAN Volume Controller manages a number of back-end disk subsystem controllers and maps the physical storage within those controllers to logical disk images that can be seen by application servers and workstations in the SAN. The SAN must be zoned in such a way that the application servers cannot see the same back-end LUNs seen by the SAN Volume Controller, preventing any possible conflict between the SAN Volume Controller and the application servers both trying to manage the same back-end LUNs. As described earlier, when an application server performs I/O to a VDisk assigned to it by the SAN Volume Controller, it can access that VDisk through either of the nodes in the I/O group. Each node can only be in one I/O group, and since each I/O group only has two nodes, the distributed redundant cache design in the SAN Volume Controller only needs to be two-way. The SAN Volume Controller I/O groups are connected to the SAN in such a way that all back-end storage and all application servers are visible to all of the I/O groups. The SAN Volume Controller I/O groups see the storage presented to the SAN by the back-end controllers as a number of disks, known as managed disks. Because the SAN Volume Controller does not attempt to provide recovery from physical disk failures within the back-end controllers, MDisks are recommended, but not necessarily required, for a RAID array. The application servers must not see the MDisks at all. Instead, they should see a number of logical disks, known as virtual disks or VDisks, which are presented to the SAN by the SAN Volume Controller. MDisks are collected into groups, known as managed disk groups (MDGs). The MDisks that are used in the creation of a particular VDisk must all come from the same MDG. Each MDisk is divided into a number of extents. The minimum extent size is 16 MB, and the maximum extent size is 2048 MB, based on the definition of its MDG. These extents are numbered sequentially from the start to the end of each MDisk. Conceptually, this is represented as shown in Figure 2-1.
VDISK 1
Figure 2-1 Extents being used to create a virtual disk
Chapter 2. IBM System Storage SAN Volume Controller overview
15
The virtualization function in the SAN Volume Controller maps the VDisks seen by the application servers to the MDisks presented by the back-end controllers. I/O traffic for a particular VDisk is, at any time, handled exclusively by the nodes of a single I/O group. Although a cluster can have several pairs of nodes, the nodes handle I/O in independent pairs. This means that the I/O capability of the SAN Volume Controller scales well (almost linearly), since additional throughput can be obtained by simply adding additional I/O groups. Figure 2-2 summarizes the various relationships that bridge the physical disks through to the virtual disks within the SAN Volume Controller architecture.
Figure 2-2 The relationship between physical and virtual disks
Virtualization mappings Several different mapping functions are provided by the SAN Volume Controller: Striped: Here a VDisk is mapped to a number of MDisks in an MDG. The extents on the VDisk are striped over the MDisks. Therefore, if the VDisk is mapped to five MDisks, then the first, sixth, eleventh (and so on) extents come from the first MDisk, the second, seventh, twelfth (and so on) extents come from the second MDisk, and so on. This is the default mapping. Sequential: Here a VDisk is mapped to a single MDisk in an MDG. There is no guarantee that sequential extents on the MDisk map to sequential extents on the VDisk, although this might be the case when the VDisk is created.
16
Implementing the IBM System Storage SAN Volume Controller V4.3
Note: There are no ordering requirements in the MDisk to VDisk extent mapping function for either striped or sequential VDisks. This means that if you examine the extents on an MDisk, it is quite possible for adjacent extents to be mapped to different VDisks. It is also quite possible for contiguous extents on the MDisk to be mapped to widely separated extents on the same VDisk, or to close-by extents on the VDisk. In addition, the position of the extents on the MDisks is not fixed by the initial mapping, and can be varied by the user performing data migration operations. Image: Image mode sets up a one-to-one mapping of extents on an MDisk to the extents on the VDisk. Because the VDisk has exactly the same extent mapping as the underlying MDisk, any data already on the disk is still accessible when migrated to a SAN Volume Controller environment. Within the SAN Volume Controller environment, the data can (optionally) be seamlessly migrated off the image mode VDisk to a striped or sequential VDisk within the same or another MDG.
Virtual Disk Copy Every VDisk is associated to at least one VDisk Copy. The VDisk itself is a logical entity whereas the VDisk Copy is a physical entity. The VDisk Copy represents the physical capacity occupied by the VDisk on the MDisks. A second copy can be created as a VDisk Copy Mirror, as shown in Figure 2-3.
Figure 2-3 VDisk and VDisk copy
2.4.3 SAN Volume Controller multipathing Each SAN Volume Controller node presents a VDisk to the SAN through multiple paths. We recommend that a VDisk be seen in the SAN by four paths. In normal operation, two nodes provide redundant paths to the same storage. This means that, depending on zoning and SAN architecture, a single host might see eight paths, to each LUN presented by the SAN Volume Controller. Because most operating systems cannot resolve multiple paths back to a single physical device, IBM provides a multipathing device driver. The multipathing driver supported by the SAN Volume Controller is the IBM Subsystem Device Driver (SDD). SDD groups all available paths to a virtual disk device and presents it to the operating system. SDD performs all the path handling and selects the active I/O path(s).
Chapter 2. IBM System Storage SAN Volume Controller overview
17
2.4.4 SAN Volume Controller logical configuration Figure 2-4 shows an example of a SAN Volume Controller configuration.
Figure 2-4 SAN Volume Controller logical view
Cluster There is a maximum of four I/O groups (node pairs) per cluster. The performance of a cluster increases almost linearly with additional I/O groups.
I/O Group Because there is a minimum of two nodes per I/O group, there is no single point failure. Even if one node breaks away, data in the cache is not lost because the cache is mirrored between the two nodes. Every virtual disk is assigned to a single I/O group. For load balancing, a virtual disk is owned by alternating nodes in a I/O Group.
Configuration notes Here are some basic characteristics and recommendations in regard to the configuration: To provide high availability, we strongly recommend that the SAN Volume Controller nodes be configured in redundant SAN fabrics. The Fibre Channel switches need to be zoned to permit the hosts to see the SAN Volume Controller nodes and the SAN Volume Controller nodes to see the RAID Controllers. The SAN Volume Controller nodes within a cluster must be able to see each other. In addition, if there are two SAN Volume Controller clusters with Metro Mirror (formerly referred to as 18
Implementing the IBM System Storage SAN Volume Controller V4.3
Peer-to-Peer Remote Copy (PPRC)) and Global Mirror services between them, zoning must be set so that all the nodes of both clusters can see each other. In addition to Fibre Channel connections, each device has an Ethernet connection for configuration and error reporting. However, only one of the nodes, the configuration node, binds an IP address to its Ethernet connection. There is also a service IP address for a cluster that can be used by any node when in “service mode”.
2.5 Software licensing SVC V4.3.0 brings several changes to the licensing scheme. However, the scheme utilized in V4.2.1 will still be used until you add additional licenses to the cluster using svctask chlicense. Refer to the SVC V4.3.0 announcement letter for the definitive statement at the following link: http://tinyurl.com/4zgam3
2.6 New with SVC V4.3 For the most up-to-date information for the SVC, go to: http://www.ibm.com/storage/support/2145
2.6.1 Space-efficient VDisk A VDisk can now be created as a space-efficient VDisk. This means that a different virtual and real capacity is defined. The real capacity defines the space actually allocated to a VDisk. The VDisk capacity shown on the host is the virtual capacity. A directory maps the virtual address space to the real address space. Directory and user data share the real capacity, as shown in Figure 2-5.
Figure 2-5 Space-efficient VDisk
This enables advanced planning for future growth, as SVC administrators can expand the physical capacity without reconfiguration or downtime on the host. However, proper storage provisioning planning is still required, regardless of whether the configuration is space-efficient or fully provisioned. In SVC, it is possible to set warnings if a VDisk runs out of its capacity. These warnings should be configured sensibly to provide adequate time to provision more physical storage, and they should not be ignored! If required, more storage can be allocated and the real capacity can gradually be extended by grains up to the virtual capacity. The virtual capacity can also be expanded, but this is a manual process.
Chapter 2. IBM System Storage SAN Volume Controller overview
19
Note: If the used capacity reaches the real capacity, then the VDisk will go offline and application I/O will fail. To get the VDisk online again, the real capacity must be expanded. To avoid exhausting the real capacity, alerts can be sent to administrators to provide space, or by using autoextend to increase the real capacity automatically. An existing VDisk / Image mode VDisk cannot be converted to a space-efficient VDisk. It can be migrated afterwards to a space-efficient VDisk.
Read and write I/O process The different scenarios of read and write processes from or to a space-efficient VDisk is explained in the following summary: Write to an unallocated region When the directory lookup indicates that the region to be written to is unallocated, the SVC allocates space and updates the directory. Space is formatted while data and directory is written to the disk. Write to an allocated region When the directory lookup indicates that the region to be written to is already allocated. Data is written to the disk in this case. Read to an unallocated region (unusual) When the directory lookup indicates that the region to be read from is unallocated, the SVC returns a buffer of 0x00s. Read to an allocated region When the directory lookup indicates that the region to be read from is already allocated, data is read from the disk in this case.
2.6.2 FlashCopy Space-efficient FlashCopy. – Copy of a space-efficient source VDisk to a space-efficient target VDisk – Copy of a fully-allocated source VDisk to a space-efficient target VDisk The background copy process does not copy unallocated regions. The incremental feature allows you to refresh FC mapping once a fully copy of the actually regions is complete.
FlashCopy now supports up to 256 target copies from a single source virtual disk. Auto-delete of consistency group. Up to 4096 FlashCopy mappings. Licensing. – Virtual capacity of space-efficient source VDisk used in calculations – Multiple target mappings from the same source consume no extra license quota.
2.6.3 VDisk mirroring Allows creation of a single VDisk with one or two copies. Copies can be in different MDGs on different disk controllers. Can be used for migration. – Migrate VDisk between MDisk groups with different extent size – Migrate from space-efficient to fully-allocated VDisk – Control copy rate of migration
20
Implementing the IBM System Storage SAN Volume Controller V4.3
2.6.4 IPv6 addresses IPv6 addresses are now supported for: – Cluster IP – Service IP – SNMP server – E-mail server – ICAT address Can have both IPv4 and IPv6 addresses for a cluster and both can used at the same time.
2.6.5 Limitations 2048 VDisk per I/O group (double of existent maximum)
2.6.6 Licensing Metro Mirror/Global Mirror is licensed by capacity. – The Metro Mirror/Global Mirror license is calculated by the capacity needed on a cluster. – If the remote copy relationship affects two clusters, the license is calculated for each cluster individually. – If space-efficient virtual disks are part of the Metro Mirror/Global Mirror relationship, the virtual capacity is used for license calculations. Note: After upgrading, the licensing system will behave as in V4.2.1. until license values are added with the svctask chlicense command.
2.6.7 Miscellaneous New software upgrade status query. – Uses the svcinfo lssoftwareupgradestatus command. – Indicated states: inactive, upgrading, downgrading, committing, and stalled. The code level is displayed on the front panel. – The node level as well as the cluster level is shown on the front panel, so you can see a code upgrade as it happens. 1625 (controller misconfiguration) errors do not call home any more. – 1626 generates warning only. – Persistent unfixed 1625 errors cause a new 1695 error that does call home. – DS3K and DS4K misconfiguration calls home with a new 1624 error. Allow more time for back-end configuration changes and re-zoning changes to occur before taking MDisk offline. Report new node error 454 for internal disk going write-only. – The service action will be disk replacement. The Master Console is now replaced by SSPC. – The V4.3. GUI console code is supported on existing master consoles.
Chapter 2. IBM System Storage SAN Volume Controller overview
21
NetApp® gateway support for V4.2 and V4.3. – Ifix is available through RPQ for V4.2 in order to disable FNR and address an issue with the NetApp driver. – No ifix is planned for V4.3. – Full support of V4.2 and FNR will be available when Data ONTAP® V7.3 becomes generally available from NetApp. – Support of V4.3 will follow in 90 days and hopefully sooner. – The NSeries gateway support will follow later when it supports Data ONTAP V7.3. Change in support for split I/O group configurations. – An RPQ is needed with a review of a solution design by ATS or development is required. – No change is required for existing working split cluster environments. – Nodes in the I/O group must be physically within 100 meters of each other. – The node and its UPS must be in the same rack. – No ISLs can be used for node-to-node communication within an I/O group. – Currently, there are no LW GBIC/SFPs for SVC HBA, so the distance is limited through RPQ to 300 meters for 4 Gbs SAN or 600 meters for a 2 Gbs connection. – Requires IBM storage for quorum disks and they must be located in a third site.
2.6.8 Additional interoperability support since SVC 4.2.1 Microsoft® Virtual Server Host Environment Progressions: – NetWare V6.5 SP2 for 4 Gb QLogic® HBA – Tru64 V5.1 B4 – OpenVMS V8.3 – RHEL V4.6, including SAN boot – RHEL V5 Itanium® (IA64) – SLES 9 SP2 – SLES 10 for System z® – VMware® ESX Server V3.5, 3i Clustering Updates: – HP-UX11iV2, PVLinks, ServiceGuard 11.18 – 2 node MSCS, x32/x64, on BladeCenter HS21 – Steeleye Lifekeeper for Linux – SUN Cluster 3.2 – SLES 10 Native Clustering – RHEL 5 Native Clustering Oracle® 10g RAC/ASM support for: – Solaris™ 10 – HPUX11iV2 – SLES 10 – RHEL 4 – RHEL 5 Veritas Storage Foundation 5.x support for: – AIX 5L V5.2 – AIX 5L V5.3 – Solaris 10 x86 – Windows® 2003 with 2 Gb HBAs
22
Implementing the IBM System Storage SAN Volume Controller V4.3
Blade Updates: – JS22 Blade Support – JS12 Blade Support – HP Blade Server support QLogic 6140 Router Brocade-McDATA interop Mode Brocade DCX Director class SAN768B Fujitsu Eternus 8000 Model 2100 Disk System
2.6.9 SVC 4.3 Interoperability enhancements Host support – Microsoft Windows 2008, including Enterprise x64 Edition, SAN boot, 32-bit support, and clustering – Microsoft Windows 2008 Enterprise Edition for Itanium-based™ systems, including SAN boot – HP-UX 11i V3 for PA-RISC and Itanium-based systems, including clustering with HP ServiceGuard – Apple Mac OS X Server 10.5.2 with ATTO Celerity FC-42ES HBA – Storage system support – Pillar Axiom Models 300 and 500
Chapter 2. IBM System Storage SAN Volume Controller overview
23
24
Implementing the IBM System Storage SAN Volume Controller V4.3
3
Chapter 3.
Planning and configuration In this chapter, we describe the steps required when planning the installation of an IBM System Storage SAN Volume Controller (SVC) in your storage network. We look at the implications for your storage network.
© Copyright IBM Corp. 2003-2008. All rights reserved.
25
3.1 General planning rules To achieve the most benefit from SAN Volume Controller (SVC), pre-installation planning should include several important steps. These steps ensure that SVC provides the best possible performance, reliability, and ease of management for your application needs. Proper configuration also helps minimize downtime by avoiding changes to SVC and the storage area network (SAN) environment to meet future growth needs. Tip: The IBM System Storage SAN Volume Controller: Planning Guide, GA32-0551, contains comprehensive information that goes into greater depth regarding the topics we discuss here. Planning the SVC requires that you follow these steps: 1. Document the number of hosts (application servers) to attach to the SVC, the traffic profile activity (read or write, sequential or random), and the performance requirements (input/output (I/O) per second). 2. Document the storage requirements and capacities. – The total back-end storage to be provisioned on the SVC. – The required virtual storage capacity (space-efficient Virtual Disk) and its associated real capacity. – The required storage capacity for local mirror copy (Virtual Disk Mirroring). – The required storage capacity for point-in-time copy (FlashCopy). – The required storage capacity for remote copy (Metro and Global Mirror). – Per host: Storage capacity, the host logical unit number (LUN) quantity, and sizes. 3. Define the local and remote SAN fabrics and clusters, if a remote copy or a secondary site are needed. 4. Define the number of clusters and the number of pairs of nodes (between 1 and 4) for each cluster. Each pair of nodes (an I/O group) is the container for the virtual disks. How many I/O groups are needed depends on the overall performance requirements. 5. Design the SAN according to the requirement for high availability and best performance. Consider the total number of ports and the bandwidth needed between the host and the SVC, the SVC and the disk subsystem, between the SVC nodes, and for the ISL between the local and remote fabric. 6. Define the managed disks (MDisks) in the disk subsystem. 7. Define the managed disk groups (MDGs). This depends on the disk subsystem in place and the data migration needs. 8. Create and re-partition the VDisks between the different I/O groups and the different MDGs in such a way as to optimize the I/O load between the hosts and the SVC. This can be an equal re-partition of all the VDisks between the different nodes, or a re-partition that takes into account the expected load from the different hosts. 9. Plan for the physical location of the equipment in the rack. 10.Determine the IP addresses for the SVC Cluster, the SVC service IP address, and SSPC (SVC console). 11. Define the number of FlashCopy required per hosts. 12.Define the cluster configuration backup and business data backup.
26
Implementing the IBM System Storage SAN Volume Controller V4.3
3.2 Physical planning There are several main factors to take into account when carrying out the physical planning of an SVC installation. The physical site must have the following characteristics: Power, cooling, and location requirements are present for the SVC and uninterruptible power supplies. SVC nodes and uninterruptible power supplies must be in the same rack as its associated SVC node. Plan for two different power sources if you have ordered a redundant AC power switch (available as an optional feature). An SVC node is one EIA unit high. Each of the uninterruptible power supplies (UPSs) that comes with SVC V4.3 is one EIA unit high; the UPS shipped with the earlier version of the SVC is two EIA units high. The SSPC (SVC console) is two EIA units high: one for the server and one for the keyboard and monitor. Other hardware devices can be in the rack, such as IBM System Storage DS4000, IBM System Storage DS6000, SAN switches, Ethernet switch, and others. Consider the maximum power rating of the rack; this must not be exceeded.
Chapter 3. Planning and configuration
27
In Figure 3-1, we show the SVC in a rack shared with other storage equipment.
Figure 3-1 SVC in its rack
28
Implementing the IBM System Storage SAN Volume Controller V4.3
3.2.1 Preparing your UPS environment Ensure that your physical site meets the installation requirements for the uninterruptible power supply (UPS). Note: The 2145 UPS-1U is a Powerware 5115 and the 2145 UPS is a Powerware 5125.
2145 UPS-1U The 2145 uninterruptible power supply-1U (2145 UPS-1U) is one EIA unit high and is shipped, and can only operate, on the following node types: SAN Volume Controller 2145-8G4 SAN Volume Controller 2145-8F2 SAN Volume Controller 2145-8F4 It was also shipped and will operate with SAN Volume Controller 2145-4F2. When you configure the 2145 UPS-1U, the voltage that is supplied to it must be 200 – 240 V, single phase. Note: The 2145 UPS-1U has an integrated circuit breaker and does not require external protection.
2145 UPS The 2145 uninterruptible power supply (2145 UPS) is two EIA units high and was only shipped with the SAN Volume Controller 2145-4F2 prior to SVC V2.1. Be aware of the following considerations when configuring the 2145 uninterruptible power supply (2145 UPS): Each 2145 UPS must be connected to a separate branch circuit. A UL-listed 15 A circuit breaker must be installed in each branch circuit that supplies power to the 2145 UPS. The voltage that is supplied to the 2145 UPS must be single phase 200 – 240 V with a supplied frequency at 50 or 60 Hz.
Heat output The maximum heat output parameters are as follows: 40 watts (135 Btu per hour) during normal operation 150 watts (510 Btu per hour) when power has failed and the UPS is supplying power to the nodes of the SAN Volume Controller Ensure that you comply with the following requirements for UPSs: If the UPS is cascaded from another UPS, the source UPS must have at least three times the capacity per phase, and the total harmonic distortion must be less than 5% with any single harmonic being less than 1%. The UPS must also have input voltage capture that has a slew rate faster than 3 Hz per second and 1 msec glitch rejections. For more detailed information, refer to the IBM System Storage SAN Volume Controller Planning Guide, GA32-0551.
Chapter 3. Planning and configuration
29
3.2.2 Physical rules The SVC must be installed in pairs to provide high availability, and each node in an I/O group must be connected to different UPSs, as shown in Figure 3-2.
Figure 3-2 Node uninterruptible power supply setup
In SVC versions prior to SVC V2.1, the Powerware 5125 UPS was shipped with the SVC; in SVC V4.2, Powerware 5115 UPS is shipped with the SVC. You can upgrade an existing SVC cluster to V4.1 and still use the UPS Powerware 5125 that was delivered with the SVC prior to V2.1. Each SVC node of an I/O group must be connected to a different UPS. Each UPS shipped with SVC V3.1, V4.1, V4.2, and V4.3 supports one node only, but each UPS in earlier versions of SVC supports up to two SVC nodes (in distinct I/O groups). Each UPS pair that supports a pair of nodes must be connected to a different power domain (if possible) to reduce the chances of input power loss. The UPSs must be installed in the lowest available position in the rack. If necessary, move lighter units toward the top. A cluster can contain up to eight SVC nodes. The power and serial connection from a node must be connected to the same UPS, otherwise the node will not boot. The 5115 and 5125 UPS can be mixed with UPSs that were supplied with earlier SVC versions, but the UPS rules above have to be followed, and SVC nodes in the same I/O group must be attached to the same type of UPSs, though not the same UPS. 8G4, 8F2, and 8F4 hardware models must be connected to 5115 UPS. They will not boot with a 5125 UPS. Important: Do not share the SVC UPS with any other devices.
30
Implementing the IBM System Storage SAN Volume Controller V4.3
Figure 3-3 shows a layout sample within a rack.
Figure 3-3 Sample rack layout
3.2.3 Cable connections Complete a cable connection table to document all of the connections required for the setup:
Nodes UPS Ethernet Fibre Channel ports SSPC (SVC Console)
Chapter 3. Planning and configuration
31
Parts of a typical planning chart are shown in Figure 3-4 and Figure 3-5.
Figure 3-4 Cable connection table
Figure 3-5 SVC (Master) Console
3.3 SAN planning and configuration SAN storage systems using the SVC can be configured with two, or up to eight, SVC nodes, arranged in an SVC cluster. These are attached to the SAN fabric, along with disk subsystems and host systems. The SAN fabric is zoned to allow the SVCs to “see” each other’s nodes and the disk subsystems, and for the hosts to “see” the SVCs. The hosts are not able to directly “see” or operate LUNs on the disk subsystems that are assigned to the SVC cluster. The SVC nodes within an SVC cluster must be able to see each other and all the storage assigned to the SVC cluster. The zoning capabilities of the SAN switch are used to create these distinct zones. The SVC in Release 4 supports 1 Gbps, 2 Gbps, or 4 Gbps Fibre Channel fabric. This depends on the hardware platform and on the switch where the SVC is connected. We recommend connecting the SVC and the disk subsystem to the switch operating at the highest speed, in an environment where you have a fabric with multiple speed switches. All SVC nodes in the SVC cluster are connected to the same SANs, and present virtual disks to the hosts. These virtual disks are created from managed disks presented by the disk subsystems. There are two distinct zones in the fabric: Host zones, to allow host ports to see and address the SVC nodes. There can be multiple host zones. See 3.3.3, “General design considerations with the SVC” on page 35 for more information. One disk zone in which the SVC nodes can see and address the LUNs presented by the disk subsystems. 32
Implementing the IBM System Storage SAN Volume Controller V4.3
Hosts are not permitted to operate on the disk subsystem LUNs directly if the LUNs are assigned to the SVC. All data transfer happens through the SVC nodes. Under some circumstances, a disk subsystem can present LUNs to both the SVC (as managed disks, which it then virtualizes to hosts) and to other hosts in the SAN. Figure 3-6 shows the data flow across the physical topology.
Host
Host
Host
Host Host Zone
SVC SVC
Disk Zone
Managed disks
SVC SVC RAID Ctrl
RAID Ctrl
RAID Ctrl
RAID Ctrl
Data Transfer
Figure 3-6 Data flows on a SVC physical topology
Logically, the two zones can be thought of as two separate logical SANs.
3.3.1 SAN definitions The following definitions are used in this section.
ISL hop An interswitch link (ISL) is a connection between two switches, and is counted as an “ISL hop.” The number of “hops” is always counted on the shortest route between two N-ports (device connections). In an SVC environment, the number of ISL hops is counted on the shortest route between the pair of nodes farthest apart. It measures distance only in terms of ISLs in the fabric.
Oversubscription Oversubscription is the ratio of the sum of the traffic on the initiator N-port connection, or connections to the traffic on the most heavily loaded ISL(s) where more than one is used between these switches. This assumes a symmetrical network, and a specific workload applied evenly from all initiators and directed evenly to all targets. A symmetrical network means that all the initiators are connected at the same level, and all the controllers are connected at the same level.
Chapter 3. Planning and configuration
33
As an example, on a 16-port switch, where there are 14 host connections going through two ISL connections, the oversubscription is 14:2, or 7:1 (14/2), with seven hosts “sharing” one ISL. In the SVC environment, each ISL link oversubscription cannot exceed six.
Redundant SAN A redundant SAN is a SAN configuration in which there is no single point of failure (SPoF), so no matter what component fails, data traffic will continue. Connectivity between the devices within the SAN is maintained, albeit possibly with degraded performance, when an error has occurred. A redundant SAN design is normally achieved by splitting the SAN into two independent counterpart SANs (two SAN fabrics), so even if one counterpart SAN is destroyed, the other counterpart SAN keeps functioning.
Counterpart SAN A counterpart SAN is a non-redundant portion of a redundant SAN. A counterpart SAN provides all the connectivity of the redundant SAN, but without the 100% redundancy. An SVC node is typically connected to a redundant SAN made out of two counterpart SANs. A counterpart SAN is often called a SAN fabric. For example, if you have only one switch, this is one fabric, or one counterpart SAN. However, if you have two switches, but they are not connected together, you have two fabrics, or two counterpart SANs, and one redundant SAN if the devices are connected to both SANs.
Local fabric Since the SVC supports remote copy, there might be significant distances between the components in the local cluster and those in the remote cluster. The local fabric is composed of those SAN components (switches, cables, and so on) that connect the components (nodes, hosts, and switches) of the local cluster together.
Remote fabric Since the SVC supports remote copy, there might be significant distances between the components in the local cluster and those in the remote cluster. The remote fabric is composed of those SAN components (switches, cables, and so on) that connect the components (nodes, hosts, and switches) of the remote cluster together.
Local and remote fabric interconnect These are the SAN components that are used to connect the local and remote fabrics. They might simply be single mode optical fibers driven by high-power GBICs or SFPs, or they might be other more sophisticated components, such as channel extenders or special SFP modules. This can be used to extend the distance to thousands of kilometers. Performance may degrade as distance increases.
Fibre Channel port logins This is the number of hosts that can see any one SVC node port. Some disk subsystems, such as the IBM DS8000, recommend limiting the number of hosts that use each port, to prevent excessive queuing at that port. Clearly, if the port fails or the path to that port fails, the host might fail over to another port and the fan-in criteria might be exceeded in this degraded mode.
Channel extender A channel extender is a device for long distance communication connecting other SAN fabric components. Generally, these can involve protocol conversion to asynchronous transfer mode (ATM), Internet Protocol (IP), or some other long distance communication protocol.
34
Implementing the IBM System Storage SAN Volume Controller V4.3
3.3.2 Fibre Channel switches, fabrics, interswitch links, and hops The local or remote fabric must not contain more than three hops in each fabric. Any configuration that causes this to be exceeded is unsupported. When a local fabric is connected to a remote fabric for Metro Mirror, the hop count between a local node and a remote node must not exceed seven. For example, node A in fabric A wishes to connect to node B in fabric B. Within fabric A, node A takes two hops to reach the ISL that connects fabric A to fabric B. The hop count is at two at this point in time. Traversing the ISL between fabric A and fabric B takes the hop count to three. Once it has reached fabric B, it takes three hops to reach node B. The hop count total is six (2+1+3), which is within our limits. Another example would be where three hops have been used within fabric A, and three hops have been used in fabric B. This means that there can only be one hop between fabric A and fabric B, otherwise the supported hop count limit of seven will be exceeded. Alternatively, where, for example, fabric A consisted of one hop, and fabric B also consisted of one hop, then this would leave up to five hops that could be used to interconnect switches between fabric A and fabric B. If multiple ISLs are available between switches, we recommend that these ISLs be trunked. Follow the switch vendor's recommendations for trunking. Note: The SVC supports the use of distance extender technology to increase the overall distance between local and remote clusters; this includes DWDM and FCIP extenders. If this extender technology involves a protocol conversion, then the local and remote fabrics should be regarded as independent fabrics, limited to three hops each. The only restriction on the interconnection between the two fabrics is the maximum latency allowed in the distance extender technology. For the latest information relating to distance limitations, visit the following Web site: http://www.ibm.com/storage/support/2145
3.3.3 General design considerations with the SVC Note: The SVC system in and of itself does not support disk redundancy, so to maintain access in the case of a disk failure requires redundancy within the disk subsystem below the SVC. To ensure high availability in SVC installations, keep the following considerations in mind when you design a SAN with the SVC.
For any SVC cluster The following general guidelines apply: An SVC node, in this case, 4F2 and 8F2, always contains two host bus adapters (HBAs), each of which has two Fibre Channel (FC) ports. If an HBA fails, this remains a valid configuration, and the node operates in degraded mode. If an HBA is physically removed from an SVC node, then the configuration is unsupported. The 8G4 and 8F4 has one HBA with four ports. All nodes in a cluster must be in the same LAN segment. This is because the nodes in the cluster must be able to assume the same cluster, or service IP, address. Make sure that the network configuration will allow any of the nodes to use these IP addresses. Chapter 3. Planning and configuration
35
To maintain application uptime in the unlikely event of an individual SVC node failing, SVC nodes are always deployed in pairs (I/O groups). If a node fails or is removed from the configuration, the remaining node operates in a degraded mode, but is still a valid configuration. The remaining node operates in write through mode, meaning the data is written directly to the disk subsystem (the cache is disabled for the write). The UPS must be in the same rack as the node it supplies, and each UPS can only have one node connected. Nodes shipped pre-SVC V2.1, with the 2145 UPS (Powerware 5125), can have two nodes connected per UPS. The Fibre Channel SAN connections between the SVC node and the switches are optical fiber. These connections can run at either 1 Gbps, 2 Gbps, or 4 Gbps, depending on your SVC and switch hardware. The 8G4 and 8F4 SVC nodes auto-negotiate the connection speed with the switch. The 4F2 and 8F2 nodes are capable of a maximum of 2 Gbps, which is determined by the cluster speed. SVC node ports must be connected to the Fibre Channel fabric only. Direct connections between SVC and host, or disk subsystem, are unsupported. We recommend that the two nodes within an I/O group be co-located, and co-location is a recommendation even for an SVC cluster (all nodes in a cluster should be located close to one another (within the same set of racks), and within the same room or adjacent rooms for ease of service and maintenance). An SVC cluster can be connected (through the SAN fabric switches) to application hosts, disk subsystems, or other SVC clusters, through short wave only optical FC connections (long wave connections are no longer supported). These can be distances of up to 150 m (short wave 4 Gbps), 300 m (short wave 2 Gbps), or 500 m (short wave 1 Gbps) between the cluster and the host, and between the cluster and the disk subsystem. Longer distances are supported between SVC clusters when using intercluster Metro or Global Mirror. IBM has tested up to 10km SFP long wave links, but we actually support whatever the manufacturer of that fabric supports in terms of long distance links. Refer to 3.5.3, “Technologies for extending the distance between two SVC clusters” on page 45 for more information. A cluster should be regarded as a single entity for disaster recovery purposes. This includes the disk subsystem that is providing the quorum disks for that cluster. This means that the cluster and the quorum disks should be co-located. We do not recommend locating the components of a single cluster in different physical locations for the purpose of disaster recovery, because this might lead to issues over maintenance, service, and quorum disk management.
For multiple SVC clusters Two SVC clusters cannot share the same disk subsystem. The consequences of sharing the same disk subsystem can result in data loss. If the same MDisk becomes visible on two different SVC clusters, then this is an error that can cause data corruption.
For the SAN fabric The following guidelines apply: The Fibre Channel switch must be zoned to permit the hosts to see the SVC nodes, and the SVC nodes to see the disk subsystems. The SVC nodes within a cluster must be able to see each other, the disk subsystems, and the front-end host HBAs. We recommend that you have a zone for the SVC nodes within a cluster, another zone between the SVC nodes and the disk subsystems, and a zone between each front-end host HBA and the SVC. Mixed speeds are permitted within the fabric, but not for intracluster communication. You can use lower speeds to extend the distance.
36
Implementing the IBM System Storage SAN Volume Controller V4.3
Uniform SVC port speed for 4F2 and 8F2 nodes: The optical fiber connections between Fibre Channel switches and all 4F2 or 8F2 SVC nodes in a cluster must run at one speed, either 1 Gbps or 2 Gbps. Operating 4F2 or 8F2 nodes with different speeds running on the node to switch connections in a single cluster is an unsupported configuration (and is impossible to configure anyway). This rule does not apply to 8F4 and 8G4 nodes because the Fibre Channel ports on these nodes auto-negotiate their speed independently of one another and can run at 1 Gbps, 2 Gbps, or 4 Gbps Each of the local or remote fabrics should not contain more than three ISL hops within each fabric. An operation with more ISLs is unsupported. When a local and a remote fabric are connected together for remote copy purposes, there should only be one ISL hop between the two SVC clusters. This means that some ISLs can be used in a cascaded switch link between local and remote clusters, provided that the local and remote cluster internal ISL count is less than three. This gives a maximum of seven ISL hops in an SVC environment with both local and remote fabrics. The switch configuration in an SVC fabric must comply with the switch manufacturer’s configuration rules. This can impose restrictions on the switch configuration. For example, a switch manufacturer might limit the number of supported switches in a SAN. Operation outside the switch manufacturer’s rules is not supported. The SAN contains only supported switches as listed on the Web at: http://www.ibm.com/storage/support/2145 Operation with other switches is unsupported. Host HBAs in dissimilar hosts or dissimilar HBAs in the same host need to be in separate zones. For example, if you have AIX and Microsoft hosts, they need to be in separate zones. Here dissimilar means that the hosts are running different operating systems or use different hardware platforms. Therefore, different levels of the same operating system are regarded as similar. This is a SAN interoperability issue rather than an SVC requirement. We recommend that the host zones contain only one initiator (HBA) each, and as many SVC node ports as you need, depending on the high availability and performance you want to have from your configuration. Note: In SVC Version 3.1 and later, the command svcinfo lsfabric generates a report that displays the connectivity between nodes and other controllers and hosts. This is particularly helpful in diagnosing SAN problems.
For the disk subsystem The following guidelines apply: In the SAN, disk subsystems are always connected to SAN switches and nothing else. Multiple connections are allowed from the redundant controllers in the disk subsystem to improve data bandwidth performance. It is not mandatory to have a connection from each redundant controller in the disk subsystem to each counterpart SAN. For example, in a DS4000 configuration in which the DS4000 contains two redundant controllers, only two controller minihubs are normally used. This means that Controller A in the DS4000 is connected to counterpart SAN A, and controller B in the DS4000 is connected to counterpart SAN B. Operation with direct connections between host and controller is unsupported. Split controller configurations are supported with certain rules and configuration guidelines. Refer to IBM System Storage SAN Volume Controller Planning Guide, GA32-0551 for more information.
Chapter 3. Planning and configuration
37
The SVC is configured to manage LUNs exported only by disk subsystems, as listed at: http://www.ibm.com/storage/support/2145 Operation with other disk subsystems is unsupported. All SVC nodes, in an SVC cluster, must be able to see the same set of disk subsystem ports on each disk subsystem controller. An operation in a mode where two nodes see a different set of ports on the same controller becomes degraded. The system logs errors requesting a repair action. This can occur if inappropriate zoning was applied to the fabric. It can also occur if inappropriate LUN masking is used. This has important implications for a disk subsystem, such as DS4000, which imposes exclusivity rules on which HBA world wide names (WWNs) a storage partition can be mapped to. It is up to you to check that the planned configuration is supported. You can find the supported hardware list at: http://www.ibm.com/storage/support/2145
For the host and application servers The following guidelines apply: Each SVC node presents a virtual disk (VDisk) to the SAN through four paths. Since in normal operation two nodes are used to provide redundant paths to the same storage, this means that a host with two HBAs can see eight paths to each LUN presented by the SVC. We suggest using zoning to limit the pathing from a minimum of two paths to the maximum available of eight paths, depending on the kind of high availability and performance you want to have in your configuration. We recommend using zoning to limit the pathing to four paths. The hosts must run a multipathing device driver to resolve this back to a single device. The multipathing driver supported and delivered by SVC is the IBM Subsystem Device Driver (SDD). Native multipath I/O (MPIO) drivers on selected hosts are supported. For operating system specific information about MPIO support, see: http://www.ibm.com/storage/support/2145 The number of paths to a VDisk from a host to the nodes in the I/O group that owns the VDisk must not exceed eight, even if this is not the maximum number of paths supported by the multi-path driver (SDD supports up to 32). To restrict the number of paths to a host VDisk, the fabric(s) should be zoned such that each host Fibre Channel port is zoned with one port from each SVC node in the I/O group that owns the VDisk. Note: The recommended number of VDisk paths is 4. If a host has multiple HBA ports, then each port should be zoned to a different set of SVC ports to maximize high availability and performance. In order to configure greater than 256 hosts, you will need to configure the host to iogrp mappings on the SVC. Each iogrp can contain a maximum of 256 hosts, so it is possible to create 1024 host objects on an eight node SVC cluster. The mappings can be configured using the svctask mkhost, svctask addhostiogrp, and svctask rmhostiogrp commands. The mappings can be viewed using the svcinfo lshostiogrp and svcinfo lsiogrphost commands. VDisks can only be mapped to a host that is associated with the I/O Group to which the VDisk belongs. The SAN Volume Controller supports connection to the Cisco MDS 9000 SAN-OS Software Release 3.2 for the Cisco MDS 9000 Family platform with the attached iSCSI hosts (in single path mode only). See the following Web site for the latest support information: http://www.ibm.com/storage/support/2145
38
Implementing the IBM System Storage SAN Volume Controller V4.3
For management The following guidelines apply: In addition to a Fibre Channel connection, each node has an Ethernet connection for configuration and error reporting. These connections are aggregated together through an Ethernet switch. All nodes in an SVC cluster must be in the same IP subnet. This is because the node in the SVC cluster must be able to assume the same SVC cluster IP address or SVC service IP address. IBM supports the option of having two master consoles to provide redundancy. Multiple master consoles can access a single cluster, but when multiple master consoles access one cluster, you cannot concurrently perform configuration and service tasks. The master console can be either preinstalled hardware or software supplied to and installed by the user.
3.3.4 Boot support The SVC supports SAN boot for AIX and Windows 2003 using MPIO, HP-UX by using PVLinks as the multipathing software for the boot device, and Solaris 9 running Veritas Volume Manager/DMP, but the SAN boot support could change from time to time, so we recommend regularly checking the following Web site: http://www.ibm.com/systems/storage/software/virtualization/svc/interop.html
3.3.5 Configuration saving We recommend that you save the configuration externally, when changes such as adding new nodes, disk subsystems, and so on have been done in the cluster.
Chapter 3. Planning and configuration
39
3.3.6 High availability SAN design and configuration rules with SVC Figure 3-7 shows a basic two node configuration. To provide high availability, the SVC should be configured in redundant SAN fabrics. Our configuration, as shown in Figure 3-7, is a redundant fabric made up of two 16-port switches.
W2K3_1 P1
AIX_270
P2
P1
P2
Linux P1
W2K3_2
P2
P1
P2
SSPC
P1 (Master Console)
P2
p4
p3
p3 0 3 4 8 9 12 14 SW11 SAN switch
SVC1 node1
14 12 9 8 4 3 0 SW21 SAN switch
1 2 5
p4 SVC1 node2
5 2 1
p2
p1
p1
p2
DS4300_A
DS4300_B
Figure 3-7 Simple two-node SVC high availability configuration
3.4 Zoning To manage the zoning in the fabric, port zoning or WWN zoning can be used. For the latest information, regularly check whether your configuration is supported at the following Web site: http://www.ibm.com/storage/support/2145 Figure 3-8 on page 41 shows an example of a host zone where each host adapter is zoned to two SVC I/O group HBA ports, one from each node in the I/O group. The numbers 11, 12, 13, and 14 are for node1 FC port 1, 2, 3, and 4. The blue zone consists of A1, 13, and 22, meaning host adapter FC port 1, node1 FC port 3, and node2 FC port 2. The position in the switch where the FC port is connected is used when making the zone definitions. For example, 13 is connected to the switch with domain ID 11 port 3, (11,3); remember that the first port in the switch starts numbering at zero.
40
Implementing the IBM System Storage SAN Volume Controller V4.3
Host
HOST-A1 Zone (21,1;11,2;11,3)
HOST-A2 Zone (22,1;12,2;12,3)
A1 A2 ID 21
A1 Fabric 1
Fabric 2 ID 11
11 22 13 24 Host port A1 and one SVC port per SVC node
ID 22
A2
11 12 13 14 SVC node1
21 12 23 14
ID 12
21 22 23 24 Host port A2 and SVC node2
Note: No storage subsystem ports!
one SVC port per SVC node
Figure 3-8 Host zoning example
With this zoning configuration, each VDisk has four paths to the host. A new host will be configured with zoning that uses the remaining, not yet used SVC nodes ports 11 , 14 , 21 , and 24 , and in this way we will perform manual load balancing on the SVC nodes in the SVC cluster. More hosts will use the same route in a round robin fashion.
3.5 Naming conventions Naming conventions in the open systems environment have always been a challenge. The challenges come from finding naming conventions that will continue to be steady as changes occur to the environment. Everyone has their own way of naming equipment in an IT infrastructure. When working in a SAN environment where an SVC cluster is installed, we recommend assigning names that help in locating and identifying equipment, and that provide information about connections so that any changes and troubleshooting are easier. One way to do this is to include site name, equipment name, and adapter information in the naming convention. As an example, in a two-site solution, site A and site B, all equipment in site A is identified by odd numbers, while all equipment at site B is identified by even numbers. Such an example could look like this: SVC1N1P1, where SVC1 is the name of the equipment, number 1 indicates that it is located in site A, N1 is the node name for the SVC cluster, and the P1 is the SVC FC port number. On site B, the name would have been SVC2N1P1. (Note that names stemming from popular culture can be amusing, but do not always give any meaningful information about what the equipment is or where it is located, which has several disadvantages.) Chapter 3. Planning and configuration
41
Figure 3-9 shows an example of a naming convention.
Naming convention
SW11
SW12
SVC1N2
SVC2N2
SVC1N1
SVC2N1
SW21
FAST1-A
DS4301_A
SW22
FAST1-B
DS4301_B
FAST2-A
DS4302_A
FAST2-B
DS4302_B
Figure 3-9 An example of a name convention in a dual SVC setup
Below, we list examples of names in an SVC cluster setup, and we document those which you can use as examples to build your own naming convention. SVC naming convention examples: SVC1N1 = SVC cluster 1, node 1 SVC2N1 = SVC cluster 2, node 1 SVC2N2P3 = SVC cluster 2, node 2, FC port 3 Disk subsystem name convention examples:
DS4301_A DS4302_B EVA301_A DS801B3A1
Here is an explanation of names for the disk subsystem: DS4301_A_1, where DS43 tells you the type of storage back end (in our example, a DS4300). DS4301_A_1, where 01 is the number of this DS4300 in your installation, and also gives you the information that it is placed at site A (1 is an odd number). DS4301_A_1, where _A is the name of the controller in the DS4300. DS4301_A_1, where _1 is the FC port number on controller A, but port number is only used in SAN zoning information, not on the SVC. Here we would only recommend DS4301 as the name. DS4302_B will then mean a DS4300, located at site B, controller B. EVA301_A will then mean a EVA3000, located at site A, controller A.
42
Implementing the IBM System Storage SAN Volume Controller V4.3
Host Fibre Channel ports FC0 and FC1: HOSTNAMExx_FC0 HOSTNAMExx_FC1 Here, xx is a number used to identify the server and give information as to its location. For example, AIX01_fc0, AIX01_fc1 gives you the type of server, what server it is, and where it is located, in this example, at site A. SAN switch names: SW11, where SW indicates a switch, the first number gives a fabric ID, and the last is the location, and even combining the last two numbers together is also useful as the domain ID of the switch. SW12 is then SAN switch 2 in fabric 1, located at site B, domain id 12. SW21 is then SAN switch 1 in fabric 2, located at site A, domain id 21. When using these kinds of names, we have a limit of 10 switches in a fabric, so if we need more switches, we will use two digits for the switch number, for example, SW101 and so on. SVC zone names: SVC1, which includes SVC1N1P1, SVC1N1P2, SVC1N1P3, SVC1N1P4, SVC1N2P1, and so on for all SVC1 nodes. SVC2, which includes all SVC2 node ports. Storage zone names: SVC1_DS4301 SVC2_DS4302 Host zone names: HOSTNAME_FC0_SVC1 HOSTNAME_FC1_SVC1 Metro or Global Mirror zone name: SVC1_SVC2 Changing the domain IDs can affect your zoning configuration and the existing setup. Therefore, you must change them first before you change your zoning information. Note: A change of the domain ID or core PID disrupts some UNIX operating systems. Make sure that you check first before you attempt this when storage is already defined and in use.
Chapter 3. Planning and configuration
43
3.5.1 Dual room high availability configuration with the SVC Figure 3-10 shows a high availability configuration of the SVC when two SVC clusters are in two different rooms/locations. We recommend this configuration for maximum availability.
Hostnn
Host01
Up to 64 hosts in each SVC cluster
SVC cluster1
SVC1N1
Host02
Hostnn
SVC cluster2
I/O group 0
SVC2N1
I/O group 0
SVC1N2
SVC2N2
SVC1N3
SVC2N3 I/O group 1
I/O group 1
SVC1N4
SVC2N4
SW11
UPS1A
UPS2A
Up to 256 ports in each fabric
UPS1B
DS4301_A
SW12
SW21
DS4301_B
UPS2B
SW22
Up to 64 controllers per cluster Site B Site A
DS4302_A
DS4302_B
Figure 3-10 High availability SAN Volume Controller cluster in a two-site configuration
3.5.2 Local and remote SAN fabrics with SVC The SVC supports both intracluster and intercluster Metro and Global Mirror. From the intracluster point of view, the only reasonable candidate for a Metro or Global Mirror operation is the other node in the same I/O group. Intercluster operation needs a pair of clusters, separated by a number of moderately high bandwidth links, which means that the bandwidth should be large enough to handle the writes from the Metro or Global Mirror process, but no reads are done over the links. Here we describe the configuration that is shown in Figure 3-10: In the SVC, the local and remote fabric interconnect supported is a single ISL hop between a switch in the local fabric and a switch in the remote fabric, if it is a single mode fiber, up to 10 km in length. Check the support Web site for any changes to this configuration at: http://www.ibm.com/storage/support/2145 SVC V4.3 supports a operation with Fibre Channel DWDM extenders and SAN Routers. The supported distances depend on the SAN fabric vendor, but the latency must be better than an 80 ms round-trip delay (40 ms one way). In Metro or Global Mirror configurations, additional zones are required that contain only the local nodes and the remote nodes. It is unsupported to create a zone that presents both the local and remote disk subsystem and either local and remote nodes or both. 44
Implementing the IBM System Storage SAN Volume Controller V4.3
3.5.3 Technologies for extending the distance between two SVC clusters Technologies for extending the distance between two SVC clusters can be divided into two categories: Fibre Channel Extenders SAN Routers
Fibre Channel extenders Fibre Channel extenders simply extend a Fibre Channel link by transmitting Fibre Channel packets across long distance links without changing the contents of those packets. Here is a list of examples: FCIP extenders implemented in CISCO MDS 9500 series switches CNT Ultranet Edge Storage Router DWDM, CWDM, and longwave SFP extenders Any Multiprotocol router (for example, Brocade Multiprotocol Routers) only when used in FCIP tunnelling mode. The maximum supported one way latency is 34 ms. Any Fibre Channel technology is supported that meets the following requirements: The one-way latency between sites must not exceed 34 ms. Note that 1 ms equates to approximately 100 km, but this will depend on the quality of circuit, the type of equipment used, and the configuration. The bandwidth between sites must be sized to meet peak workload requirements while maintaining the maximum latency of 34 ms. If the link between sites is configured with redundancy so that it can tolerate single failure, then the link must be sized so that the bandwidth and the latency statements continue to hold true even during such single failure conditions. A channel extender can be used only for inter-cluster links (intra-cluster is not supported). The entire configuration must be tested with the expected peak workload. The configuration is tested to simulate a failure of a primary site, and eventually a failback from the secondary site to the primary site (to test recovery procedures). The configuration is tested to confirm that any failover mechanism in the inter-cluster links interoperate with the SVC. Particular attention must be taken for the compatibility between switches if they are from different vendors, and also between the switches and extender. Measurements about latency and bandwidth must be made during installation and the records must be kept. Testing should be repeated before and after any significant change in the infrastructure providing the inter-cluster links.
SAN Routers SAN Routers extend the scope of a SAN by providing “virtual nPorts” on two or more SANs. The router arranges it so that traffic at one virtual nPort is propagated to the other virtual nPort, but the two Fibre Channel fabrics are independent of one another. Thus, nPorts on each of the fabrics cannot directly log into each other.
Chapter 3. Planning and configuration
45
At the time of writing, some of the systems IBM supported are: McDATA 1620 and 2640: These are supported up to a one way latency of 10 ms. Note that 1 ms equates to approximately 100 km to 150 km, but this will depend on the type of equipment used, links, and the configuration. Cisco MDS 9000 series Inter VSAN Routing: The use of Inter VSAN Routing in the configuration using MD 9000 series switches is supported with latency up to 10 ms. Latency is approximately about 1 ms for 100-150 km, but this will depend on the type of equipment used, links, and the configuration. You can find the detailed supported list of inter-cluster extenders and routers on the Web at: http://www-03.ibm.com/systems/storage/software/virtualization/svc/interop.html Important: It is the latency that IBM will support, not the distance. The above distances are provided for illustrative purposes only.
3.6 SVC disk subsystem planning This section covers planning SVC disk subsystem components.
3.6.1 Block virtualization The managed disk group (MDG) is at the center of the many-to-many relationship between managed disks and virtual disks. It acts as a container into which managed disks contribute chunks of disk blocks, known as extents, and from which virtual disks consume these extents of storage. MDisks in the SVC are LUNs assigned from the underlying disk subsystems to the SVC, and can be either managed or unmanaged. A managed MDisk is an MDisk assigned to an MDG. MDGs are collections of managed disks. A managed disk is contained within exactly one MDG. An SVC supports up to 128 MDGs. There is no limit to the number of virtual disks that can be in an MDG other than the limit per cluster. MDGs are collections of virtual disks. Under normal circumstances, a virtual disk is associated with exactly one MDG. The exception to this is when a virtual disk is migrated, or mirrored, between MDGs.
3.6.2 MDGs, I/O groups, virtual disks, and managed disks Figure 3-11 on page 47 shows three disk subsystems that were configured to provide a number of LUNs. In SVC terminology, each of these logical units (LUNs) is a managed disk. Disk subsystem A contains two managed disks, known as M1 and M2. Disk subsystem B contains managed disks M3 and M4. Disk subsystem C contains managed disks M5 and M6.
46
Implementing the IBM System Storage SAN Volume Controller V4.3
I/O Group RS
I/O Group PQ SVC Node
SVC Node
P
Q
SVC Node
Some other I/O group
Some other I/O group
V3
V2
V1
Managed Disk Group Y
Managed Disk Group X
M1
SVC Node S
R
M2
Disk Subsystem A
M3
M4
Disk Subsystem B
V5
V4
V6
V7
Managed Disk Group Z
M5
M6
Disk Subsystem C
Figure 3-11 Disk relationships
Figure 3-11 also shows three MDGs: X, Y, and Z. Each managed disk is contained within a single MDG, and that one MDG can span disk subsystems. SVC supports an arbitrary relationship between disk subsystems and MDGs. The MDG simply contains a collection of managed disks from the set of available disk subsystems. Note: For LUNS to be a part of same MDG, we recommend that all the LUNs that are part of that MDG come from: Disk subsystems having similar characteristic, for example, IOPS, cache, throughput, and so on. Hard disks having the same characteristics, for example, type, speed, and RPM. For further details, refer to SAN Volume Controller: Best Practices and Performance Guidelines, SG24-7521. Figure 3-11 also shows virtual disks numbered V1 to V7. Each virtual disk is contained entirely within an MDG. This is a normal situation. The only exception to this is during migration. Virtual disks are also members of another collection, namely I/O groups. Figure 3-11 shows two I/O groups named PQ and RS. An I/O group contains an arbitrary set of virtual disks and exactly two SVC nodes (unless one has failed). The I/O group defines which nodes support I/O request from hosts.
Chapter 3. Planning and configuration
47
There is no fixed relationship between I/O groups and MDGs. An individual virtual disk is normally a member of one MDG and one I/O group: The MDG defines virtual disk is carved using which managed disks, which are coming from the underlying disk subsystem. The I/O group defines which SVC nodes provide I/O access to the virtual disk.
3.6.3 Extents A virtual disk occupies an integer number of extents. Its length does not need to be an integer multiple of the extent size, but must be an integer multiple of the block size. Any space left over between the last logical block in the virtual disk and the end of the last extent in the virtual disk is unused. You can define a VDisk with the smallest granularity of 512 bytes (a block). However, an entire extent is reserved even if it is partially used. An extent is never shared between virtual disks. Each extent on a managed disk is contained with at the most one virtual disk. Free extents are associated with no virtual disk. SVC supports extent sizes of 16, 32, 64, 128, 256, 512, 1024, and 2048 MB. The extent size is a property of the MDG, which is set when the MDG is created. It cannot be changed and all managed disks, which are contained in the MDG, have the same extent size, so all virtual disks associated with the MDG must also have the same extent size. Table 3-1 shows the relationship between the extent size and the maximum capacity of the cluster. Table 3-1 Extent size and maximum cluster capacities Extent size
Maximum cluster capacity
16 MB
64 TB
32 MB
128 TB
64 MB
256 TB
128 MB
512 TB
256 MB
1 PB
512 MB
2 PB
1024 MB
4 PB
2048 MB
8 PB
3.6.4 Image mode virtual disk Image mode provides a direct block-for-block translation from the managed disk to the virtual disk with no virtualization. This mode is intended to allow virtualization of managed disks, which already contain data that was written directly, not through an SVC node from a pre-virtualized disk subsystem. When an image mode virtual disk is created, it directly corresponds to the managed disk it is created from. This allows you to insert an SVC into the data path of an existing storage environment with minimal downtime. After the SVC is inserted into the data path using image mode, you can use the migration facilities to migrate the data to managed mode and rearrange the data while an application is accessing the data.
48
Implementing the IBM System Storage SAN Volume Controller V4.3
When you create an image mode virtual disk, the managed disk specified must not be a member of an MDG. The managed disk is made a member of the specified MDG as a result of the creation of the image mode virtual disk. Image mode provides direct mapping from managed disk to virtual disk. You can think of it as a property of both virtual disks and managed disks. The capacity specified must be less than or equal to the size of the managed disk. If it is less than the size of the managed disk, then the unused space in the managed disk is not available for use in any other virtual disk. There is no facility to specify an offset. Therefore, logical block address (LBA) “N” on the resulting image mode virtual disk maps directly to LBA “N” on the image mode managed disk. Image mode virtual disks have a minimum size of one block (512 bytes) and always occupy at least one extent. Image mode managed disks are members of an MDG, but do not contribute free extents to the pool of free extents. Therefore, an image mode managed disk can have at most one virtual disk associated with it.
3.6.5 Managed mode virtual disk Disks operating in managed mode provide a full set of virtualization functions. Within an MDG, the SVC supports an arbitrary relationship between extents on (managed mode) virtual disks and extents on managed disks. Subject to the constraint that each managed disk extent is contained in at most one virtual disk, each virtual disk extent maps to exactly one managed disk extent (except when in the progress of migrating).
Chapter 3. Planning and configuration
49
Figure 3-12 shows virtual disk V, which is made up of a number of extents. Each of these extents is mapped to an extent on one of the managed disks A, B, or C. The mapping table stores the details of this indirection. You can see that some of the managed disk extents are unused, that is, there is no virtual disk extent that maps to them. These unused extents are available for creating new virtual disks, migration, expansion, and so on.
Figure 3-12 Simple view of block virtualization
Creating managed mode virtual disks When a virtual disk is created, the SVC needs to know the policy to apply to create the initial assignment of managed disk extents to virtual disk extents. The supported policies are listed in the following sections. These policies are only used for the creation of a new virtual disk. After the virtual disk is created, the policy has no effect and is not considered when making decisions during migration operations.
Striped When a virtual disk is created using a striped policy, its extents are allocated from the specified ordered list of managed disks. The allocation algorithm starts with the first managed disk in the ordered list and attempts to allocate an extent from it, then moves to the next disk, and so on, for each managed disk in turn. If the specified managed disk has no free extents, then it misses its turn, and the turn passes to the next managed disk in the list. When the end of the list is reached, the algorithm loops back to the first disk in the list. Allocation proceeds until all the required extents have been allocated. When selecting which extent to allocate from the chosen managed disk, the policy followed is as described in 3.6.7, “Extent allocation and size rules” on page 55. This allocation policy leads to coarse grained striping. The granularity of the striping is at the extent level. This coarse grained striping is unlikely to result in large bandwidth for sequential transfers, but is
50
Implementing the IBM System Storage SAN Volume Controller V4.3
likely to spread the workload caused by random small transactions across the managed disks from which the extents are allocated. Wide striping increases the probability that the data on the virtual disk will be lost due to the failure of one of the managed disks across which the virtual disk is striped. It is acceptable for the list to contain only one disk. In this case, extents are allocated from a single disk as described in 3.6.7, “Extent allocation and size rules” on page 55. Contrast this with the allocation scheme for the sequential policy.
Sequential When a virtual disk is created using a sequential policy, its extents are allocated from a single specified managed disk. The SVC searches for regions of the target managed disk that contain free extents that are sequential so that the region is large enough to allocate the virtual disk from completely sequential extents. If it finds more than one such region, it chooses the smallest region that satisfies this condition. If it does not find any suitable regions, creation of the virtual disk fails.
Cache modes and cache disabled VDisks Prior to SVC V3.1, enabling any copy services function in a RAID array controller for a LUN that was being virtualized by SVC was not supported because the behavior of the write-back cache in SVC would have led to data corruption. With the advent of cache disabled VDisks, it becomes possible to enable copy services in the underlying RAID array controller for LUNs that are virtualized by SVC. Note: Wherever possible, we recommend using SVC copy services in preference to the underlying controller copy services.
Using underlying controller remote copy with SVC cache disabled VDisks Where synchronous or asynchronous remote copy is used in the underlying storage controller, the controller LUNS at both the source and destination must be mapped through the SVC as image mode disks with the SVC cache disabled. Note that of course it is possible to access either the source or the target of the remote copy from a host directly, rather than through the SVC. The SVC copy services can be usefully employed with the image mode virtual disk representing the primary of the controller remote copy relationship, but it would not make sense to use SVC copy services with the VDisk at the secondary site. This is because the SVC does not see the data flowing to this LUN through the controller.
Using underlying controller FlashCopy with SVC cache disabled VDisks Where FlashCopy is used in the underlying storage controller, the controller LUNS for both the source and target must be mapped through the SVC as image mode disks with the SVC cache disabled. Note that it is possible to access either the source or the target of the FlashCopy from a host directly, rather than through SVC. The SVC copy services can be used with the VDisk representing the source for the controller FlashCopy relationship, but it would not make sense to use SVC copy services with the VDisk representing the controller FlashCopy target. This is because the SVC does not see the data flowing to this LUN through the controller.
Controlling copy services on the underlying storage controller Where a storage controller has a copy services interface that is accessed over an IP connection (out-of-band), there will be little difference in the way that the copy services are controlled when the SVC is added between the controller and the servers. Where a storage controller has a copy services interface that is accessed in-band, it might still be possible to control the copy services from the hosts through the in-band interface. This should be addressed on a controller by controller basis.
Chapter 3. Planning and configuration
51
As stated, with SVC Version 3.1, you can choose if you want read and write operations to be stored in cache by specifying a cache mode. You must specify the cache mode when you create the VDisk. After the VDisk is created, you cannot change the cache mode. Table 3-2 describes two different types of cache modes for a VDisk. Table 3-2 Cache mode parameters Cache modes
Description
readwrite
All read and write I/O operations that are performed by the VDisk are stored in cache. This is the default cache mode for all VDisks.
none
All read and write I/O operations that are performed by the VDisk are not stored in cache.
3.6.6 Space-efficient Virtual Disk In release 4.3.0, a new feature has been introduced known as space-efficient VDisk (SEV). This feature enables the VDisk size (virtual capacity) to be larger than the space occupied by VDisk Copy 0 (zero), physical capacity, on MDisk. The real capacity defines the amount space that is actually allocated to the VDisk, and the virtual capacity defines the size of the VDisk as it appears to the host. The mapping between the virtual address space to the real address space is implemented using the directory map. This directory map is implemented as B-Tree and stored on back-end storage. This is shown in Figure 3-13 on page 53. The SVC consumes less than 1% of used capacity to store the metadata for the directory map. Important: In an event where the real capacity is fully utilized, either due to a lack of available physical storage on auto-expand, or because of a disk not marked for auto-expand, and it has not been expanded manually, this will result in the VDisk entering an offline state. This is not a data integrity or loss issue, as the data is maintained in the SVC cache until additional storage space is provisioned. However, the SVC’s cache cannot be treated or used as a backup mechanism, and the necessary steps for the provisioning of additional storage space should be immediately implemented.
52
Implementing the IBM System Storage SAN Volume Controller V4.3
Space-Efficient VDisk Host Server
Virtual Capacity
Managed Disk Group
Directory
Controller
Real Capacity
Figure 3-13 Space-efficient VDisk mapping
The VDisk is allocated sufficient extents to make up the VDisk real capacity. The host’s write I/O on to the VDisk at any LBA utilizes the space in the real capacity, resulting in an increase of the used capacity. Again, if the real capacity is fully utilized, then the VDisk goes offline and application I/O to that VDisk fails. To make this VDisk online again, the administrator must provision for more storage to expand the real capacity of the VDisk.
Chapter 3. Planning and configuration
53
Figure 3-14 shows the relationship between virtual capacity, real capacity, and used capacity.
Virtual Capacity 2GB Virtual LBA 0
Virtual LBA Max
Real Capacity 1GB Real LBA Max
Real LBA 0
Free Capacity
400MB
Used Capacity
600MB
Warning Level for Used Capacity
800MB
Figure 3-14 Space-efficient VDisk’s virtual capacity, real capacity, and used capacity relationship
SVC provides the ability to avoid exhaustion of real capacity by allowing alerts to be sent to the administrator for additional capacity or for automatically increasing the real capacity. The allocation unit for the real capacity is known as the grain size and can be 32 KB, 64 KB, 128 KB, or 256 KB. The space is allocated when data is first returned to a virtual LBA. Once allocated, the space cannot be de-allocated and it is the responsibility of the volume manager to reuse the space that is freed up due to deletion of data or files on that VDisk. The space-efficient VDisk is a feature that helps an application or group of users that require a higher storage space capacity, but the actual utilization is much lower. For example, some of the applications require a higher amount of free space at the time of installation, but it is not actually utilized until data is written to that part of the disk. This gives the storage administrator the ability to meet the higher demand of storage space, but at the same time efficiently utilize the valuable storage space, and support a pay as you grow strategy for disk subsystems. However, space-efficient VDisk requires proper planning, as shown in “Space-efficient virtual disk considerations” on page 58.
54
Implementing the IBM System Storage SAN Volume Controller V4.3
3.6.7 Extent allocation and size rules Migration operations and some of the virtualization operations require the allocation of a specific number of extents from a specific set of managed disks. The algorithm used to achieve this task is described in the following sections.
Choosing the managed disk to allocate from Where the set of managed disks to allocate extents from contains more than one disk, extents are allocated from managed disks in a round robin fashion. If a managed disk has no free extents when its turn arrives, then its turn is missed and the round robin moves to the next managed disk in the set that has a free extent. As the algorithm progresses, disks with no free extents on the previous pass of the round robin are queried for free extents on each turn of the algorithm in case extents become free.
Choosing the extent to allocate from a specific managed disk When an extent is to be allocated from a specific managed disk, the allocation policy is to allocate the next free extent from a list of free extents held by the SVC cluster for the specific managed disk.
Size of extent If you want to migrate a VDisk from one MDisk group to another MDisk group, the extent size must be the same between the two MDisk groups. Because of this, it can be useful to set a common extent size for all the MDisk groups. A value of 32 MB (corresponding to a maximum cluster capacity of 128 TB) or 64 MB (corresponding to a maximum cluster capacity of 256 TB) can be the best trade-off between performance and capacity. If you need to migrate a VDisk from one MDisk group to another MDisk group that has another extent size, you need to use Metro or Global Mirror so the VDisks have to belong to the same I/O group if they are in the same SVC cluster.
3.6.8 MDisk group planning There are several guidelines or rules that you must follow when creating an MDisk group.
Number of MDGs The number of MDisk groups depends on the following factors: The need for image mode virtual disks (data migration) The need for managed mode virtual disks with sequential policy The models of the disk subsystem controller (disk subsystem with cache or without, DS4000, ESS, and so on) that have different properties of performance, availability, response time, and so on The models of the hard disks in the disk subsystem controller (hard disk type, RPM, speed, and so on) that have different properties on performance, availability, response time, and so on. It is possible to have a common MDG for the SVC cluster. However, a virtual disk (VDisk) is offline if any managed disk in the MDG is offline, even if that managed disk does not contribute any extents to the virtual disk in question, or the managed Disk has no allocated extents. The more managed disks there are in an MDG, the more the VDisk (host LUN) is striped and the better the performance is.
Chapter 3. Planning and configuration
55
We recommend that you: Create at least one separate MDG for all the image mode virtual disks. Create one separate MDG for each array (or RAID) type presented from a disk subsystem, or one separate MDG for each subsystem when the RAID protection is the same for all the subsystems or types of the hard disks on the disk subsystem. So the MDGs are characterized by performance, RAID level, reliability, vendor, and so on. Keep in mind that more MDG granularity reduces the possibility to take VDisks offline due to MDisk problems or subsystem maintenance procedures, but this level of granularity increases the management activity required. Note: It could be wise to keep each disk subsystem in a separate MDisk group. This prevents a failure in storage subsystem A from affecting VDisks in an MDisk group from storage subsystem B. If a VDisk is composed of MDisks from both A and B, then a failure in either A or B causes the VDisk to be unavailable. Name them in such a way that it is easy for you (when you create a virtual disk) to associate a virtual disk with an MDG that has the appropriate level of performance and reliability needed, for example, pool1_high_perf_high_rela, pool2_low_perf_low_rela, mDisk_grp_ESS1, mDisk_grp_DS40002, mDisk_grp_raid10, and so on.
Creating a managed disk First, you need to create the logical disks (LUNs) in your disk subsystem that will be made available as MDisks in the SVC. We recommend that you use the maximum LUN size to be presented as an MDisk. The discovery of the managed disk is automatically done by the SVC. The managed disk is in unmanaged mode until you include it in an MDG. You need at least one managed disk for the support of the quorum disk used in the cluster. All SVC nodes must have access at any time to all the managed disks. The size of the managed disk can be up to 2 TB. You can use some common sizes for all the managed disks (16 GB or 32 GB). This helps simplify things and ensures that, as much as possible, all the MDisks are used in the striping process for a managed disk with striped policy. If you have three managed disks, two of 4 GB and one of 210 GB, then only the disk of 210 GB is used in the striping process. If you want to migrate data from an existing back-end LUN, you do not have to create an MDisk (LUNs in the disk subsystem) of that size, since you already have that LUN. Simply make an image mode vdisk without specifying the size, which will then default to the size of the existing LUN or unmanaged MDisk.
3.6.9 Planning a virtual disk An individual virtual disk is a member of one MDG and one I/O group.
Selecting MDGs There is only one question you might want to ask regarding the selection of MDGs: From which MDisk group should I create my VDisk? The answer to this question is that you need to keep in mind that an individual virtual disk is a member of one MDG and one I/O group: The MDG defines which managed disks provided by the disk subsystem make up the virtual disk. The I/O group (two nodes make an I/O group) defines which SVC nodes provide I/O access to the virtual disk.
56
Implementing the IBM System Storage SAN Volume Controller V4.3
Note: There is no fixed relationship between I/O groups and MDGs. Therefore, you could define the virtual disks using the following considerations: Optimize the performance between the hosts and SVC by distributing the VDisks between the different nodes of the SVC cluster. This means spreading the load equally on the nodes in the SVC cluster. Get the level of performance, reliability, and capacity you require by using the MDG that corresponds to your needs (you can access any MDG from any node), that is, choose the MDG that fulfils the demands for your VDisk, with respect to performance, reliability, and capacity.
I/O handling and offline conditions For a virtual disk to be online, all managed disks in the MDG or MDGs associated with the virtual disk must be online. A virtual disk is offline if any managed disk in the MDG is offline, even if that managed disk does not contribute any extents to the virtual disk in question, or the managed disk has no allocated extents. Note: Normally, a virtual disk is associated with just one MDG. However, for the duration of a migration between MDGs, the virtual disk is associated with two MDGs. In this case, the offline rules apply to both MDGs for the duration of the migration only. Referring back to Figure 3-11 on page 47, this means that if managed disk M1 is taken offline by disk subsystem A, virtual disks V1 and V2 are taken offline by the SVC. This notion of offline and online is managed on a node basis. Therefore, if a condition arises that causes one SVC node to see a managed disk offline, then the affected virtual disks are taken offline on that node only, but may still be online at the other node. For example, refer again to Figure 3-11 on page 47. If the SAN connection between disk subsystem B and SVC node P were to fail, then node P would lose contact with managed disks M3 and M4. Since M3 is in MDG X and M4 is in MDG Y, this takes all the virtual disks in MDGs X and Y on node P offline. Therefore, hosts accessing node P see virtual disk V2 go offline. Hosts accessing V2 through node Q continue to see the virtual disk as online. When using SDD, the paths to node P show offline, while the paths to node Q show online, and the host still has access to the virtual disks.
I/O group considerations When you create a VDisk, it is associated to one node of an I/O group. By default, every time you create a new VDisk, it is associated to the next node using a round robin algorithm. For example, you might have four hosts (host1, host2, host3, and host4) with 100 VDisks for each host of the same size with the same level of I/O activity, and a four node (two I/O groups) cluster. The result is 100 VDisks on each node (25 VDisks from host1, 25 VDisks from host2, and so on). You can specify a preferred access node. This is the node through which you send I/O to the VDisk instead of using the round robin algorithm. For example, in one host with four VDisks (VD1, VD2, VD3, and VD4). VD1 and VD3 have a high level of I/O activity. VD2 and VD4 have a low level of I/O activity. If you use the round robin algorithm, VD1 and VD3 are on the same node 1 of the I/O group, and VD2 and VD4 are on the same node 2 of the I/O group. To avoid this, use the preferred node feature to specify VD1 on node 1 and VD3 on node 2.
Chapter 3. Planning and configuration
57
A virtual disk is defined for an I/O group, which provides the following benefits: The VDisk is “exported” by the two nodes of the I/O group to the host through eight paths (four paths for each node). We use zoning to limit it to four paths from each node. Each write is copied into the cache of the two nodes before acknowledgment is sent to the host. Even if you have eight paths for each virtual disk, all I/O traffic flows only towards one node (the preferred node). Therefore, only four paths are really used by SDD. The other four are used only in case of a failure of the preferred node or when CCU is running. Before you create a virtual disk, you can check the amount of space that is available in the MDG. You can determine the free capacity for an MDisk or an MDisk group using the svcinfo lsmdiskgrp <MDG_Name> command.
Creating image mode virtual disks Use image mode virtual disks when a managed disk already has data on it, from a pre-virtualized disk subsystem. When an image mode virtual disk is created, it directly corresponds to the managed disk from which it is created. Therefore, virtual disk LBA x = managed disk LBA x. The capacity of image mode VDisks defaults to the capacity of the supplied MDisk, but can be reduced if desired. When you create an image mode disk, the managed disk must have a mode of unmanaged and therefore does not belong to any MDG. A capacity of 0 is not allowed. Image mode virtual disks can be created in sizes with a minimum granularity of 512 bytes, and must be at least one block (512 bytes) in size. The SVC can reserve an integer number of extents to hold the image mode disk. It effectively rounds up its size to the nearest whole number of extents.
Creating managed mode virtual disks with sequential policy When creating a managed mode virtual disk with sequential policy, you must use a managed disk containing free extents that are sequential and of a size that is equal or greater than the size of the virtual disk you want to create. It may be the case that there are sufficient extents available on the managed disk, but that there is no contiguous block large enough to satisfy the request.
Space-efficient virtual disk considerations The space-efficient VDisk is great feature, but it requires proper planning for effective utilization of it. While creating the space-efficient volume, it is necessary to understand the utilization patterns by the applications or group users accessing this volume. Items such as the actual size of the data, rate of creation of new data, modifying or deletion of existing data, and so on all need to be taken into consideration. Depending on the initial size for the real capacity, the grain size and warning level can be set. If a disk goes offline, either through a lack of available physical storage on auto-expand, or because a disk marked as non-expand has not been expanded, then there is a danger of data being left in the cache until some storage is made available. This is not a data integrity or data loss issue, but you should not be relying on the SVC cache as a backup storage mechanism.
58
Implementing the IBM System Storage SAN Volume Controller V4.3
Recommendation: We highly recommend to keeping Warning Level on Used Capacity such that it provides adequate time for provision of more physical storage. Warnings should not be ignored by administrator. Use the auto-expand feature of SEV. The grain size allocation unit for the real capacity in VDisk can be set as 32 KB, 64 KB, 128 KB, or 256 KB. A smaller grain size utilizes space more effectively, but it results in a larger directory map, which may reduce performance. Once allocated, space can never be just de-allocated. It is the responsibility of the volume manager to reuse the space that is freed up due to deletion of data or files on that VDisk. Therefore, to achieve the effective utilization of storage space, it is necessary for the administrator to perform regular maintenance on volume manager for SEV. In the case of running a data erasure type of application that writes to every LBA of the VDisk during the destruction of the data on the disk, consult IBM Technical Support prior to running such an application.
3.6.10 Planning for operations on virtual disks You can perform several operations on virtual disks.
Modifying a virtual disk You can change the I/O group with which a virtual disk is associated. This requires a flush of the cache within the nodes in the current I/O group to ensure that all data is written to disk. I/O should be suspended at the host level before you perform this operation, and an SDD reset or cfgmgr restart is likely to be needed if the target WWNNs change.
Expanding a virtual disk A virtual disk can be expanded. The granularity of expansion is one block (512 bytes). If the expansion requires the allocation of additional extents, then these are allocated to the virtual disk from the managed disks specified using the algorithm described in 3.6.7, “Extent allocation and size rules” on page 55. Expanding a virtual disk using the sequential policy forces the virtualization policy to be changed to striped. Image mode virtual disks cannot be expanded. They must first be migrated to managed mode. Warning: Not all volume managers can tolerate the expansion of a virtual disk. A reboot or remount of the disk might be needed to use the additional space.
Reducing a virtual disk A virtual disk can be shrunk. The granularity of shrinking is one block (512 bytes). If the shrink operation allows extents to be freed, then these are returned to the pool of free extents for allocation by later virtualization and migration operations. Image mode virtual disks cannot be reduced in size. They must first be migrated to managed mode.
Chapter 3. Planning and configuration
59
Warning: Not all volume managers can tolerate a virtual disk being reduced in size. You must be cautious and know where data resides, or data loss can occur.
Deleting a virtual disk A virtual disk can be deleted. When a virtual disk is deleted, all host mappings are deleted and any cached read or write data is discarded. Also, any FlashCopy mappings or Metro or Global Mirror relationships in which the disk is participating in are also deleted. If the virtual disk was operating in managed mode, then the extents are returned to the pool of free extents for allocation by later virtualization operations. If the virtual disk was an image mode virtual disk, deleting the virtual disk causes the managed disk to be ejected from the MDG. The mode of the managed disk is returned to “unmanaged” mode. This makes the delete operation the inverse of the create operation for image mode disks.
Migration This facility allows the mapping of virtual disk extents to managed disk extents to be changed, without interrupting a host’s access to that virtual disk. You can perform this for any virtual disk managed by the SVC. You can use this for:
Redistributing workload within a cluster across the disk subsystem Moving workload onto newly installed storage Moving workload off old or failing storage, ahead of decommissioning it Moving workload to rebalance a changed workload Migrating data from an older disk subsystem to SVC managed storage Migrating data from one disk subsystem to another VDisk Migration (from MDisk Group to another MDisk Group)
As mentioned above, you might want to migrate VDisks from one set of MDisks to another to retire an old disk subsystem, or to have better balance performance across your virtualized environment, or simply to migrate data into the SVC environment transparently using image mode. A sufficient amount of free space or extents must be available in the target MDG. If insufficient extents are available within your target MDG, you will receive an error message. Make sure the source and target MDG have the same extent size. The optional threads parameter allows you to assign a priority to the migration process. The default is 4, which is the highest priority setting. However, if you want the process to take a lower priority over other types of I/O, you can specify 3, 2, or 1.
Migrate a VDisk to an image mode VDisk To migrate a VDisk to an image mode VDisk, the following rules apply: The destination MDisk must be greater than or equal to the size of the VDisk. The MDisk specified as the target must be in an unmanaged state. Regardless of the mode that the VDisk starts in, it is reported as managed mode during the migration. Both of the MDisks involved are reported as being Image mode during the migration. If the migration is interrupted by a cluster recovery or by a cache problem, then the migration will resume after the recovery completes. For further details about the migration facility, see Chapter 14, “Migration to and from the SAN Volume Controller” on page 741.
60
Implementing the IBM System Storage SAN Volume Controller V4.3
Also with V4.3.0, there is one more possibility that exists for migration, from space-efficient to non-space-efficient VDisk and vice-versa. This migration is not supported with the CLI. The SVC will carry out migration, or copying, of data on a per-block basis. In the current release, the SVC is unable to identify the zero data block per OS perspective. Hence, migrating from a non space-efficient VDisk to a space-efficient VDisk requires the target VDisk to be the same size as the source and target VDisk that is being 100% utilized.
Quality of service on VDisk You can set the I/O governing rate, which is a cap on the amount of I/O that is accepted for this virtual disk. You can set it in terms of I/O per second or MBs per second. By default, no I/O governing is set when a virtual disk is created. An I/O threshold is expressed as a number of I/Os, or a number of MBs, over a minute. The threshold is evenly divided between all SVC nodes that service the VDisk, that is, between the nodes that form the I/O Group of which the VDisk is a member. The algorithm operates two levels of policing: while I/O is under threshold, and when threshold is reached. While a VDisk on each SVC node receives I/O at a rate below the governed level, then no governing is performed. A check is made every minute that the VDisk on each node is continuing to receive I/O below the threshold level. If this check shows that the host has exceeded its threshold on one or more nodes, then policing begins for new I/Os. While policing is in force: A threshold quantity is calculated for a one-second period. I/Os are counted over a period of a second. If I/Os are received in excess of the one-second threshold quantity on any node in the I/O, those I/Os are grouped and later I/Os are pended. When the second expires, a new threshold quantity is established, and any pended I/Os are re-driven under the new threshold quantity. If a host stays within its one second threshold quantity on all nodes in the I/O group for a period of one minute, then the policing is relaxed, and monitoring takes place over the one-minute period as it was before the threshold was reached.
3.6.11 Host considerations In this section, we discuss creating a host.
Port masking You can use a port mask to control the node target ports that a host can access. This satisfies two requirements: As part of a security policy, to limit the set of WWPNs that are able to obtain access to any VDisks through a given SVC port. As part of a scheme to limit the number of logins with mapped VDisks visible to a host multi-pathing driver (like SDD) and thus limit the number of host objects configured without resorting to switch zoning. The port mask is an optional parameter of the svctask mkhost and chhost commands. The port mask is four binary bits. Valid mask values range from 0000 (no ports enabled) to 1111 (all ports enabled). For example, a mask of 0011 enables port 1 and port 2. The default value is 1111 (all ports enabled). Chapter 3. Planning and configuration
61
Standard and persistent reserve On 4.1 code and above, you can use the svctask rmvdiskhostmap command to remove standard and persistent reservations that a host has on the VDisk.
3.6.12 Quorum disks A quorum disk is used to resolve tie-breaker situations, when the “voting set” of nodes disagree on the current cluster state. The voting set is an overview of the SVC cluster configuration running at a given point in time, and is the set of nodes and quorum disk that are responsible for the integrity of the SVC cluster. On cluster creation, the voting set consists of a single node with a unique ID of 1, which was used to create the cluster. When nodes are integrated into the SVC cluster, they get added to the voting set, and when a node is removed from the SVC cluster, it will also be removed from the voting set. A failed node is considered a removed node, and is removed from the voting set. When MDisks are added to the SVC cluster, it checks the MDisk to see if it can be used as a quorum disk. If the MDisk fulfils the demands, the SVC will assign the three first MDisks as quorum candidates, and one of them is selected as the active quorum disk. If possible, the SVC will place the quorum candidates on different disk subsystems. Once the quorum disk has been selected, however, no attempt is made to ensure that the other quorum candidates are presented through different disk subsystems. When the set of quorum disk candidates has been chosen, it is fixed. A new quorum disk candidate will only be chosen if: The administrator requests that a specific MDisk becomes a quorum disk using the svctask setquorum command. An MDisk that is a quorum disk is deleted from an MDG. An MDisk that is a quorum disk changes to image mode. An MDisk will not be replaced as a quorum disk candidate simply because it is offline. The cluster must contain at least half of its nodes in the voting set to function. A tie-breaker situation can occur if exactly half the nodes in the cluster fail at the same time, or if the cluster is divided so that exactly half the nodes in the cluster cannot communicate with the other half. For example, in a cluster of four nodes, if any two nodes fail at the same time, or any two cannot communicate with the other two, a tie-breaker condition exists and must be resolved. To resolve the tie-breaker condition, a quorum disk is used. The cluster automatically chooses three managed disks to be quorum disks. One of these disks is used to settle a tie-breaker condition. If a tie-breaker condition occurs, the first half of the cluster to access of the quorum disk after the split has occurred locks the disk and continues to operate. The other side stops. This action prevents both sides from becoming inconsistent with each other. In a two-site solution, we recommend using two SVC clusters, one at each site, and mirroring the data by using either host mirror software/functions or by using SVC Metro or Global Mirror. In a two-site solution with only one SVC cluster, you can have a situation where you will lose access to the data, and is not allowed. For example, in a four node SVC cluster, with two nodes at each location, the quorum will only be located at one of the sites, and if that site “dies,” the remaining two nodes cannot get access to the quorum disk, and will also shut down. As a result, the entire SVC cluster is shut down, even though only one site is out. The same applies in a two-node SVC cluster, if you put the two nodes in different locations or rooms. Important: A cluster should be regarded as a single entity for disaster recovery purposes. This means that the cluster and the quorum disk should be co-located.
62
Implementing the IBM System Storage SAN Volume Controller V4.3
Figure 3-15 and Figure 3-16 on page 64 show two different scenarios with respect to quorum disk and cluster co-location.
Bad One IO Group Design, if HA is key issue
I/O group 0
SVC1N1
SVC1N2
SVC cluster
UPS1A
DS4301_A
SW11
SW12
SW21
SW22
UPS1B
D S4302_A
DS4301_B
DS4302_B
Site B
Site A
=Quorum disk Figure 3-15 Bad scenario for quorum disk and cluster co-location
Chapter 3. Planning and configuration
63
HA Design
SVC cluster1
SVC1N1
SVC cluster2
I/O group 0
SVC2N1
I/O group 0
SVC1N2
SVC2N2
SVC1N3
SVC2N3 I/O group 1
I/O group 1
SVC1N4
SVC2N4
UPS1
DS4301_A
SW11
SW12
SW21
SW22
UPS2
DS4302_A
DS4301_B
DS4302_B
Site B
Site A
= Quorum disk Figure 3-16 Correct HA scenario for quorum disk and cluster co-location
3.6.13 Expanding an SVC cluster configuration You can expand an SVC cluster configuration.
Adding a node to a cluster SVC clusters of up to 8 nodes are supported. You can easily add new nodes, add new hosts, or redistribute workload.
Adding a new disk controller to an MDisk group MDisk groups can span disk subsystems. We recommend that you do not do this. Each MDisk group should, in normal circumstances, comprise disks from one disk subsystem. See 3.6.2, “MDGs, I/O groups, virtual disks, and managed disks” on page 46 for more information.
VDisk size increase and decrease The SVC allows you to increase and decrease the size of VDisks. Not all operating systems allow this scenario. See “Expanding a virtual disk” on page 59 for more information.
64
Implementing the IBM System Storage SAN Volume Controller V4.3
3.7 Storage subsystem planning For a list of the maximum configurations, go to: http://www.ibm.com/storage/support/2145 In the configuration shown in Figure 3-17, the disk subsystem presents a LUN to the SVC and another LUN to host B. The SVC presents the VDisks created from the MDisk to the host A. Since the disk subsystem is a DS4000, host B would have RDAC installed to support the direct attachment, and SDD is installed on host A to support the attachment to the SVC. This is a supported configuration.
Host A SDD
Host B RDAC
VDisk SVC
SAN
MDisk
LUN
Array
Array
RAID Controller Figure 3-17 Disk subsystem shared
Chapter 3. Planning and configuration
65
With the ESS, you can attach directly to an ESS LUN and to a VDisk from the SVC that comes from the ESS, as long as the same LUN is not assigned to both the host and the SVC. The host uses SDD to access the LUNs presented from both the ESS and the SVC. This is a supported configuration and is shown in Figure 3-18.
Host A SDD
VDisk SVC
SAN
MDisk
LUN
Array
Array
DS8000 Figure 3-18 Host connected to ESS and SVC
In the configuration shown in Figure 3-19 on page 67, the host needs to have both the RDAC driver installed for access to the DS4000 and SDD installed to access the SVC. The SVC supports the use of the IBM SDD, native Multi-path I/O (MPIO) drivers on selected operating systems, and some other operating system specific software. We recommend having different HBAs for RDAC and SDD with proper zoning. Check for supported configurations on the Web at: http://www.ibm.com/support/docview.wss?uid=ssg1S1003278
66
Implementing the IBM System Storage SAN Volume Controller V4.3
Host A SDD/RDAC
VDisk
SAN
SVC
MDisk
LUN
Array
Array DS4000
Figure 3-19 RDAC and SDD drivers installed
3.7.1 Adding DS8000 storage to the SVC Perform the following steps to add DS8000 storage using the DS command-line Interface: 1. Go to the Web site and check the prerequisites: http://www.ibm.com/storage/support/2145 2. You might need to upgrade the microcode level of the DS8000 to support this attachment. 3. Before the MDisks can be presented to the SVC, add the DS8000 ports to the storage zone to all SVC node ports. 4. Sign on to the DSCLI of DS8000 (Example 3-1). Example 3-1 Logon to DSCLI C:\Program Files\IBM\dscli\profile>dscli Date/Time: June 5, 2007 11:49:12 PM IST IBM DSCLI Version: 5.2.200.381 IBM.2107-7520331
Chapter 3. Planning and configuration
67
Display a list of array sites to verify if any of them are available (Example 3-2). Example 3-2 lsarraysite command dscli> lsarraysite -dev IBM.2107-7520331 Date/Time: June 6, 2007 12:01:27 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 arsite DA Pair dkcap (10^9B) State Array ============================================= S1 2 73.0 Assigned A0 S2 2 73.0 Assigned A1 S3 2 73.0 Assigned A2 S4 2 73.0 Assigned A3 S5 2 73.0 Assigned A38 S6 2 73.0 Unassigned S7 2 73.0 Unassigned S8 2 73.0 Assigned A23 S9 0 73.0 Assigned A4 S10 0 73.0 Assigned A10 S11 0 73.0 Assigned A16 S12 0 73.0 Assigned A22 S13 0 73.0 Assigned A28 S14 0 73.0 Assigned A35 S15 0 73.0 Assigned A17 S16 0 73.0 Unassigned S17 7 73.0 Assigned A32
5. Make a new array (Example 3-3). Example 3-3 mkarray command dscli> mkarray -raidtype 5 -arsite S6 -dev IBM.2107-7520331 Date/Time: June 6, 2007 12:04:10 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 CMUC00004I mkarray: Array A39 successfully created. dscli>
6. Display the new array (Example 3-4). Example 3-4 lsarray command (Truncated output) dscli> lsarray -dev IBM.2107-7520331 Date/Time: June 6, 2007 12:05:54 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 Array State Data RAIDtype arsite Rank DA Pair DDMcap (10^9B) ==================================================================== A0 Assigned Normal 5 (6+P+S) S1 R0 2 73.0 A1 Assigned Normal 5 (6+P+S) S2 R1 2 73.0 A2 Assigned Normal 5 (6+P+S) S3 R2 2 73.0 A3 Assigned Normal 5 (6+P+S) S4 R3 2 73.0 . . . A37 Assigned A38 Assigned A39 Unassigned A42 Assigned A43 Assigned dscli>
68
Normal Normal Normal Normal Normal
5 5 5 5 5
(7+P) (7+P) (7+P) (6+P+S) (6+P+S)
S22 S5 S6 S35 S36
R37 R38 R42 R43
7 2 2 5 5
Implementing the IBM System Storage SAN Volume Controller V4.3
73.0 73.0 73.0 73.0 73.0
7. Create the ranks (Example 3-5). Example 3-5 Create the rank mkrank dscli> mkrank -stgtype fb -array A39 -dev IBM.2107-7520331 Date/Time: June 6, 2007 12:22:57 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 CMUC00007I mkrank: Rank R39 successfully created. dscli>.
8. Display the ranks (Example 3-6). A truncated output is shown for brevity. Example 3-6 lsrank dscli> lsrank -dev IBM.2107-7520331 Date/Time: June 6, 2007 12:24:03 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-752033 ID Group State datastate Array RAIDtype extpoolID stgtype =============================================================== R0 0 Normal Normal A0 5 P0 ckd R1 1 Normal Normal A1 5 P1 ckd R2 0 Normal Normal A2 5 P2 fb R3 1 Normal Normal A3 5 P3 fb R4 0 Normal Normal A4 5 P4 ckd R5 1 Normal Normal A5 5 P15 fb R6 0 Normal Normal A6 5 P6 ckd R7 1 Normal Normal A7 5 P5 ckd . . . R35 R36 R37 R38 R39 R42 R43 dscli>
1 0 1
Unassigned Unassigned Normal Unassigned Unassigned Normal Normal
Normal Normal Normal Normal Normal Normal Normal
A35 A36 A37 A38 A39 A42 A43
5 5 5 5 5 5 5
P37 P42 P43
ckd ckd ckd fb fb fb fb
9. Create the extpool (Example 3-7). Example 3-7 Creating the extpool dscli> mkextpool -dev IBM.2107-7520331 -rankgrp 1 -stgtype fb ITSOSVC_extpool Date/Time: June 6, 2007 12:29:27 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 CMUC00000I mkextpool: Extent pool P22 successfully created.
10.Display the newly created extpool (Example 3-8). Example 3-8 Display the newly created extpool (truncated output) dscli> lsextpool -dev IBM.2107-7520331 Date/Time: June 6, 2007 12:32:03 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 Name ID stgtype rankgrp status availstor(2^30B) %allocated available reserved numvols =========================================================================================== ===== ITSOSVC_extpool P22 fb 0 below 0 100 0 0 0 dscli>
Chapter 3. Planning and configuration
69
11.Assign the rank to the newly created extpool (Example 3-9). Example 3-9 Assigning the rank dscli> chrank -dev IBM.2107-7520331 -extpool P22 R39 Date/Time: June 6, 2007 12:52:56 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 CMUC00008I chrank: Rank R39 successfully modified. dscli>
12.Display the extpool status (Example 3-10). Example 3-10 Displaying the extpool status (truncated output) dscli> lsextpool -dev IBM.2107-7520331 Date/Time: June 6, 2007 12:56:26 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 Name ID stgtype rankgrp status availstor (2^30B) %allocated available reserved numvols =========================================================================================== =========== ITSOSVC_extpool 0 0
P22 fb
0 below
452
0
452
13.Create an FB volume with IDs 2100, 2101, 2102, and 2103 (Example 3-11). Example 3-11 mkfbvol dscli> mkfbvol -dev IBM.2107-7520331 -extpool P22 -cap 1 2100-2103 Date/Time: June 6, 2007 1:05:04 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 CMUC00025I mkfbvol: FB volume 2100 successfully created. CMUC00025I mkfbvol: FB volume 2101 successfully created. CMUC00025I mkfbvol: FB volume 2102 successfully created. CMUC00025I mkfbvol: FB volume 2103 successfully created.
14.Create the volume group (Example 3-12). Example 3-12 mkvolgrp dscli> mkvolgrp -dev IBM.2107-7520331 -volume 2100-2103 -type scsimap256 SVC_VG Date/Time: June 6, 2007 1:10:28 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 CMUC00030I mkvolgrp: Volume group V6 successfully created.
15.Display the list of volume groups (Example 3-13). Example 3-13 lsvolgrp dscli> lsvolgrp -dev IBM.2107-7520331 Date/Time: June 6, 2007 1:11:29 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 Name ID Type ======================================= CraigAIX V0 SCSI Mask SVC4_VolGroup V1 SCSI Mask WorkshopVG_GM V2 SCSI Mask Dan_Systemi_Lpar2 V3 OS400 Mask WorkshopVG_MM V4 SCSI Mask Dan_pSeries_Lpar3 V5 SCSI Mask SVC_VG V6 SCSI Map 256 Dan_Systemi_Lpar3 V9 OS400 Mask All CKD V10 FICON/ESCON All
70
Implementing the IBM System Storage SAN Volume Controller V4.3
All Fixed Block-512 V20 SCSI All All Fixed Block-520 V30 OS400 All dscli>
16.Display the volume group you created with its volumes (Example 3-14). Example 3-14 showvolgrp dscli> showvolgrp -dev IBM.2107-7520331 V6 Date/Time: June 6, 2007 1:14:16 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 Name SVC_VG ID V6 Type SCSI Map 256 Vols -2100 2101 2102 2103
17.Configure the I/O port (Example 3-15). Example 3-15 setioport dscli> setioport -dev IBM.2107-7520331 -topology SCSI-FCP I0103 Date/Time: June 6, 2007 1:19:11 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 CMUC00011I setioport: I/O Port I0103 successfully configured. dscli>.
18.Display the list of I/O ports (Example 3-16). Example 3-16 lsioport dscli> lsioport -dev IBM.2107-7520331 Date/Time: June 6, 2007 1:20:03 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 ID WWPN State Type topo portgrp =============================================================== I0010 5005076303010194 Online Fibre Channel-SW SCSI-FCP 0 I0011 5005076303014194 Online Fibre Channel-SW SCSI-FCP 0 I0012 5005076303018194 Online Fibre Channel-SW SCSI-FCP 0 I0013 500507630301C194 Online Fibre Channel-SW SCSI-FCP 0 I0030 5005076303030194 Online Fibre Channel-LW SCSI-FCP 0 I0031 5005076303034194 Online Fibre Channel-LW SCSI-FCP 0 I0032 5005076303038194 Online Fibre Channel-LW SCSI-FCP 0 I0033 500507630303C194 Online Fibre Channel-LW SCSI-FCP 0 I0040 5005076303040194 Online Fibre Channel-LW SCSI-FCP 0 I0041 5005076303044194 Online Fibre Channel-LW SCSI-FCP 0 I0042 5005076303048194 Online Fibre Channel-LW SCSI-FCP 0 I0043 500507630304C194 Online Fibre Channel-LW SCSI-FCP 0 I0100 5005076303080194 Online Fibre Channel-LW SCSI-FCP 0 I0101 5005076303084194 Online Fibre Channel-LW SCSI-FCP 0 I0102 5005076303088194 Online Fibre Channel-LW SCSI-FCP 0 I0103 500507630308C194 Online Fibre Channel-LW SCSI-FCP 0
19.Configure the SVC attachment to the DS8000. You must define eight host connections in a two-node environment, one for each single Fibre Channel port of the SVC (Example 3-17). Example 3-17 mkhostconnect dscli> mkhostconnect -dev IBM.2107-7520331 -wwname 5005076801302C49 -profile "IBM SAN Volume Controller" -volgrp V1 -ioport all SVC4_Node1 Date/Time: June 6, 2007 1:18:12 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 CMUC00012I mkhostconnect: Host connection 0003 successfully created.
Chapter 3. Planning and configuration
71
20.Repeat these steps for all FC ports of your SVC. 21.Finally, display the host connections defined (Example 3-18). Example 3-18 lshostconnect dscli> lshostconnect -dev IBM.2107-7520331 Date/Time: June 6, 2007 1:21:11 AM IST IBM DSCLI Version: 5.2.200.381 DS: IBM.2107-7520331 Name ID WWPN HostType Profile portgrp volgrpID ESSIOport =========================================================================================== ====== SVC4_Node1 0011 5005076801302C49 SVC San Volume Controller 0 V1 all SVC4_Node1 0012 5005076801402C49 SVC San Volume Controller 0 V1 all SVC4_Node1 0013 5005076801202C49 SVC San Volume Controller 0 V1 all SVC4_Node1 0014 5005076801102C49 SVC San Volume Controller 0 V1 all SVC4_Node2 0015 500507680140376F SVC San Volume Controller 0 V1 all SVC4_Node2 0016 500507680130376F SVC San Volume Controller 0 V1 all SVC4_Node2 0017 500507680120376F SVC San Volume Controller 0 V1 all SVC4_Node2 0018 500507680110376F SVC San Volume Controller 0 V1 all
3.7.2 Adding DS4000 storage to the SVC To add DS4000 storage to the SVC, follow these steps: 1. Check the prerequisites on the Web at: http://www.ibm.com/storage/support/2145 2. Check the supported firmware levels and configurations before you connect to the SVC. You can see the firmware version for the DS4000 in the Storage Manager by choosing Storage Subsystem → View profile, as shown in Figure 3-20 on page 73, Figure 3-21 on page 73, and Figure 3-22 on page 74.
72
Implementing the IBM System Storage SAN Volume Controller V4.3
Figure 3-20 Where to find the Storage Subsystem Profile
Figure 3-21 The Storage Subsystem Profile showing the firmware version
Chapter 3. Planning and configuration
73
Figure 3-22 DS4000 mappings view
3. We defined one storage partition in the DS4000 with all of the SVC node ports defined, and one host partition with eight ports. See Figure 3-23.
Figure 3-23 Shows the array created
Figure 3-24 on page 75 shows the host type for the SVC.
74
Implementing the IBM System Storage SAN Volume Controller V4.3
Figure 3-24 Host type for storage partition
Figure 3-25 shows the port mapping. Now the volumes are presented to the SVC on both ports. You can find the MDisks on the SVC and rename them to a unique name identifying the origin. Consider the example of F1_Array_LUN - DS4000 number 1, _array number_LUN number. You can add the MDisk to an MDisk group called DS4000_14830, for example.
Figure 3-25 Port mapping
Chapter 3. Planning and configuration
75
3.7.3 LUN layout When assigning LUNs from your disk subsystem to the SVC, assign the LUN to all ports in the SVC, and spread the LUNs in the disk subsystem, so you get the best performance and reliability. In the DS4000, you should have an equal number of arrays and LUNs, equally spread on the two controllers. After having assigned spare disks in the DS4000, define your arrays with RAID protection, and create as few LUNs as you can in the array. If possible, make the LUNs the same size, so you can utilize the full capacity when striping VDisks on the SVC. In an MDG where the VDisks are all striped, we recommend that all the MDisks be the same size if possible, so you can utilize the full capacity when striping VDisks on the SVC.
76
Implementing the IBM System Storage SAN Volume Controller V4.3
4
Chapter 4.
Performance and capacity planning While storage virtualization with SVC improves flexibility and provides simpler management of a storage infrastructure, it can also provide a substantial performance advantage for a variety of workloads. The SVC’s caching capability and its ability to stripe VDisks across multiple disk arrays is the reason why performance improvement is significant when implemented with midrange disk subsystems, since this technology is often only provided with high-end enterprise disk subsystems. Note: Technically, almost all storage controllers provide both striping (RAID 1 or RAID 10) and some form of caching. The real advantage is the degree to which you can stripe the data, that is, across all MDisks in a group, and therefore have the maximum number of spindles active at once. The caching is a secondary reason and the point is that the SVC provides additional caching above what midrange controllers provide (usually a couple of GB), whereas enterprise systems have much larger caches. To ensure the desired performance and capacity of your storage infrastructure, we recommend that you do a performance and capacity analysis to reveal the business requirements of your storage environment. When this is done, you can use the guidelines in this chapter to design a solution that meets the business requirements.
© Copyright IBM Corp. 2003-2008. All rights reserved.
77
4.1 Performance considerations When discussing performance for a system, it always comes down to identifying the bottleneck, and thereby the limiting factor of a given system. At the same time, you must take into consideration the component for whose workload you do identify a limiting factor, since it might not be the same component that is identified as the limiting factor for different workloads. When designing a storage infrastructure using SVC, or implementing SVC in an existing storage infrastructure, you must therefore take into consideration the performance and capacity of the SAN, the disk subsystems, the SVC, and the known/expected workload.
4.1.1 SAN The SVC now has four models: 4F2, 8F2, 8F4, and 8G4. The first number indicates the memory size; the last number represents the maximum fabric speed. All of them can connect to 1 Gbps, 2 Gbps, or 4 Gbps switches. From a performance point of view, it is better to connect the SVC to 4 Gbps switches. There are some guidelines about designing a IBM TotalStorage: SAN Product, Design, and Optimization Guide, SG24-6384. It is available in PDF format at: http://www.redbooks.ibm.com/redbooks/pdfs/sg246384.pdf Correct zoning on the SAN switch will bring security (zoning is not designed to provide security, but is often used to provide a level of security) and performance together. We recommend to implement a dual HBA approach at the host to access the SVC, and we recommend that you use the zoning shown in Figure 4-1 on page 79. During host discovery, both the SVC host path definitions and the switch zoning affect the number of occurrences of a given VDisk that the host will identify. We recommend that the number of occurrences should be kept to four or fewer, because increasing the number to more than four does not improve either performance or reliability, and may cause more time and trouble for the administrator responsible for the OS.
78
Implementing the IBM System Storage SAN Volume Controller V4.3
Host
HOST-A1 Zone (21,1;11,2;11,3)
HOST-A2 Zone (22,1;12,2;12,3)
A1 A2 ID 21
A1
A2
Fabric 1
Fabric 2
11 22 13 24 Host port A1 and one SVC port per SVC node
ID 22
11 12 13 14 SVC node1
ID 11
21 12 23 14
ID 12
21 22 23 24 Host port A2 and SVC node2
one SVC port per SVC node
Figure 4-1 Host side zone recommendation
The configuration shown in Figure 4-1 is best for many tens of VDisks and where only one path will be used actively (the other is a failover path and SDD will only send to the preferred node). With this design, the number of VDisk occurrences that the host sees during a discovery will be equal to four (the VDisk will be seen twice through each side of the dual fabric, once on the preferred node and once on the alternate node). For smaller numbers of VDisks with greater throughput requirements, more than one HBA should be mapped again to a single SVC node port each.
4.1.2 Disk subsystem There are a number of IBM Redbooks that cover the topic of tuning IBM disk subsystems for performance: IBM TotalStorage DS8000 Series: Performance Monitoring and Tuning, SG24-7146 http://www.redbooks.ibm.com/redbooks/pdfs/sg247146.pdf IBM TotalStorage DS6000 Series: Performance Monitoring and Tuning, SG24-7145 http://www.redbooks.ibm.com/redbooks/pdfs/sg247145.pdf DS4000 Best Practices and Performance Tuning Guide, SG24-6363 http://www.redbooks.ibm.com/redbooks/pdfs/sg246363.pdf
Chapter 4. Performance and capacity planning
79
From the performance perspective, there are a few guidelines in connecting with SVC: 1. Connect all storage ports to the switch and zone them to all the SVC ports. You zone all ports on the disk back-end storage to all ports on the SVC nodes in a cluster. And you must also make sure to configure the storage subsystem LUN Masking settings to map all LUNs to all the SVC WWPNs in the cluster. The SVC is designed to handle large quantities of multiple paths from back-end storage. 2. Using as many as possible 15K RPM disks will improve performance considerably. 3. Creating one LUN per array will help in a sequential workload environment. 4. There are some settings that should be changed on the storage subsystem when working with the SVC. For example, with the DS4000, we recommended setting the start/stop flushing parameter to 80%/80%, set the Cache block size to 4 KB, and set the segment size according to the DataBase application behavior when working with the SVC. For further reference, see the IBM System Storage SAN Volume Controller: Software Installation and Configuration Guide, which is available at: http://www-1.ibm.com/support/docview.wss?rs=591&uid=ssg1S7002156 In most cases, the SVC will be able to improve the performance, especially on middle-low end disk subsystems or older disk subsystems with slow controllers or uncached disk systems. This improvement happens because: The SVC has the capability to stripe across disk arrays and it can do so across the entire set of supported physical disk resources. The SVC has a 4 to 8 GB cache and it has an advanced caching mechanism. The SVC’s large cache and advanced cache management algorithms also allow it to improve upon the performance of many types of underlying disk technologies. SVC’s capability to manage, in the background, the destaging operations incurred by writes (while still supporting full data integrity), has the potential to be particularly important in achieving good database performance. Depending upon the size, age, and technology level of the disk storage system, the total cache available in the SVC may be larger, smaller, or about the same as that associated with the disk storage. Since hits can occur in either the upper (SVC) or the lower (disk controller) level of the overall system, the system as a whole can take advantage of the larger amount of cache wherever it is located. Thus, if the storage control level of cache has the greater capacity, hits to this cache should be expected to occur, in addition to hits in the SVC cache. Also, regardless of their relative capacities, both levels of cache will tend to play an important role in allowing sequentially organized data to flow smoothly through the system. SVC cannot increase the throughput potential of the underlying disks in all cases. Its ability to do so depends upon both the underlying storage technology, as well as the degree to which the workload exhibits “hot spots” or sensitivity to cache size or cache algorithms. IBM SAN Volume Controller 4.2.1 Cache Partitioning, REDP-4426, shows the SVC’s cache partitioning capability: http://www.redbooks.ibm.com/abstracts/redp4426.html?Open
4.1.3 SVC The SVC cluster is scalable up to eight nodes, and the performance is almost linear when adding more nodes into an SVC cluster, until it becomes limited by other components in the storage infrastructure. While virtualization with the SVC provides a great deal of flexibility, it does not diminish the necessity to have a SAN and disk subsystems that can deliver the 80
Implementing the IBM System Storage SAN Volume Controller V4.3
desired performance. Essentially, SVC performance improvements are gained by having as many MDisks as possible, therefore creating a greater level of concurrent I/O to the back end without overloading a single disk or array. In the following sections, we discuss the performance of the SVC and assume that there are no bottlenecks in the SAN or on the disk subsystem.
Latency Latency is the delay added to the response time for an I/O operation. All in-band storage virtualization solutions add some latency to cache miss I/Os. This is not a unique characteristic of the SVC. However, the SVC latency is very low. For a 4 KB read operation, the SVC introduces approximately 60 µs (microseconds, millionths of a second) of additional delay. Considering that a typical cache miss response time is approximately 10 ms (milliseconds, thousandths of a second), the delay typically caused by the SVC is negligible (less than 1% of total response time). In the real world, the effect of latency is normally even less. All writes to the SVC are cache hits, so they add no latency. Because of the advanced cache algorithm in the SVC, many reads are also cache hits. Only cache miss I/Os add latency.
Performance increases SVC V4.2 posted the highest results in the SPC-1 and SPC-2 disk subsystem storage performance test by the Storage Performance Council. SPC-1 consists of a single workload designed to demonstrate the performance of a storage subsystem while performing the typical functions of business critical applications. Those applications are characterized by predominately random I/O operations and require both queries as well as update operations. Examples of those types of applications include OLTP, database operations, and mail server implementations. SPC-2 consists of three distinct workloads designed to demonstrate the performance of a storage subsystem during the execution of business critical applications that require the large-scale, sequential movement of data. Those applications are characterized predominately by large I/Os organized into one or more concurrent sequential patterns. Examples of those types of applications include large file processing, large database queries, and Video on Demand. The SPC Web site can be found at: http://www.storageperformance.org Ideas International keeps a list of results ranked by throughput at: http://www.ideasinternational.com/ In IBM internally, there is a white paper about SVC V4.2 performance that is available. Contact your IBM pre-sales contact to get it by going to: http://w3-03.ibm.com/sales/support/ShowDoc.wss?docid=B552897U42160U19&infotype=SK& infosubtype=M0 There is also an IBM Redbooks publication available that covers more detailed information about SVC performance called SAN Volume Controller: Best Practices and Performance Guidelines, SG24-7521, found at: http://www.redbooks.ibm.com/abstracts/sg247521.html?Open
Chapter 4. Performance and capacity planning
81
The areas to be considered when planning a storage infrastructure using SVC, or implementing the SVC in an existing storage infrastructure, are listed in the following sections.
Determine the number of SVC clusters If you plan to deploy the SVC in a geographically dispersed environment, for example, a dual site design for disaster recovery reasons, it is essential to use two SVC clusters. Due to the design of the SVC cluster, we do not recommend splitting I/O groups geographically (for the nodes in an I/O group), since this will not provide the resiliency needed for a disaster recovery solution. There is a specific limit on the distance between nodes due to latency; this can cause the cluster to “lease expire,” which will stop I/O. Note: IBM recommends that you use two or more cluster configurations for all production disaster recovery systems. Intracluster Metro Mirror consumes more cluster resources than intercluster Metro Mirror.
Guidelines in creating the managed disk group Here are some guidelines to follow when creating a managed disk group (MDG): All the managed disks in a single MDG should have the same (or similar) performance characteristics. If you mix managed disks with different performance characteristics, VDisks might exhibit uneven performance where I/O to different portions of the VDisk perform differently. The overall performance will be that of the slowest member; all I/O will be slowed to the slowest MDisk due to the way the cache works. For high availability, easier maintenance, and troubleshooting reasons, we recommend that you do not include multiple disk subsystems in the same MDG, since the failure of one disk subsystem will make the MDG go offline, and thereby all VDisks belonging to the MDG will go offline. For example, if you have IBM DS4000 and EMC CLARiiON CX3-80, put the DS4000 MDisks in one MDG and the CX3-80 MDisks in another MDG. If there are less than 20 MDisks from one back-end storage, put them into one MDG. Using a smaller extent size can help balance the workload onto different MDisks, different LUNs, and different physical disks, and this will help the performance. Note: The extent size does not have a great impact on the performance of an SVC installation, and the most important consideration is to be consistent across MDGs. This means that you should use the same extent size in all MDGs within an SVC cluster to avoid limitations when migrating VDisks from one MDG to another. Another consideration is the storage capacity that this SVC cluster will potentially manage. For example, a 16 MB extent size cluster can manage a 64 TB storage capacity, while a 512 MB extent size cluster can manage 2 PB.
Guidelines in creating a VDisk Here are some guidelines to follow when creating a VDisk: Always choose stripe as the VDisk mode. We recommend that you stripe a VDisk among all the MDisks in an MDG, unless you need to ensure that data is provided from one and only one disk. The striping will balance I/O across all the managed disks in the MDG, which tends to optimize overall performance and helps to reduce hot spots.
82
Implementing the IBM System Storage SAN Volume Controller V4.3
Note: There is an exception here. If you are setting up for a purely sequential workload, for example, large transfer video on demand type applications, then making SVC VDisks sequential will allow SVC and the back-end MDisks to detect the sequential nature of the workload, and thus double prefetch will kick in. This, however, will limit the ultimate performance to that data that can be read off the single MDisk providing the VDisk. This should only be used when a mid to high back-end controller is presenting the MDisk and it has a large cache with a good prefetch algorithm, for example, DS6000/8000 type disk subsystems. Set the cache mode to read/write whenever possible. By default, the SVC will alternate VDisks between the nodes of an I/O group in the order the VDisks are created, and this normally produces good results. This will generally balance the load well across both nodes of an I/O group. In cases where VDisks vary greatly in size, or where the I/O load to different VDisks varies greatly, then you can use the -node parameter when creating VDisks (or the equivalent graphical user interface parameter) to specify which node of an I/O group should be the preferred path, in order to balance the workload evenly for the I/O group.
Guidelines for creating space-efficient VDisks Space-efficient VDisks require more I/Os because of the increased directory access. For a 100% write workload the impact can be, in a worst case scenario, up to 50% (one directory I/O for every user I/O). For reads, the impact depends on whether the data is in the cache, or if the metadata directory is in the cache. This causes more CPU processing, so the performance per I/O group will be lower. However, the directory is two-way write-back cached (just like the SVC fastwrite cache), so some applications will perform better. Note: To get the best space-efficient VDisk performance, use striping to spread space-efficient VDisks across many back-end disks. Be careful when using space-efficient VDisks and virtualization to reduce the number of disks required, as this is likely to cause a performance problem.
I/O queue depth handling in large SANs The purpose of discussing I/O queue depth is to avoid situations where an SVC node reaches its maximum number of queued commands. Note: Unless you have an SVC configuration close to the maximum supported, and all your hosts are simultaneously busy, it is unlikely that you will encounter problems with I/O queue depth limitations for your SVC. As a guideline, you should consult your IBM representative in case your calculation shows that the I/O queue limit is less than 20; see “Homogeneous queue depth calculation” on page 85.
Chapter 4. Performance and capacity planning
83
The queuing of tasks consumes internal resources in the SVC node. Each SVC node can handle 10,000 concurrent commands, distributed across all hosts and all VDisks. Mechanisms are provided to ensure correct operation in the event that the I/O queue is full. Each host port will be guaranteed to be able to queue a single command on an SVC node (this is per node, not per VDisk). I/O governing can be used to restrict the I/Os a host can submit. If the SVC runs out of resources to queue an I/O that it has received, the algorithm shown in Figure 4-2 takes effect to handle the situation when the maximum of queued commands is reached.
Unable to enqueue an I/O
Has initiator already consumed its specially reserved command on this node
Yes
1. Set “Unit Attention - Commands cleared by another initiator” on the LUN for that initiator
2.
Discard the command
No
Has the initiator at least one task queued for that LUN on this port
If a Unit Attention is already set, then the command is simply discarded
Yes
Return Task Set Full status, using the specially reserved command
No
The initiator has no tasks queued for that LUN on this port
Then
Return Check Condition “Unit Attention - Commands aborted by another initiator” to the received command, using the specially reserved command
Figure 4-2 I/O queue depth algorithm
This algorithm allows the SVC to discard commands, and to give a valid reason to the host as to why this has happened. This algorithm is also used when internal recoveries within the SVC node mean that the SVC is unable to start new host I/Os immediately, and the SVC consequently runs out of resources to queue all the I/Os that are received. Unfortunately, many host operating systems do not have helpful recovery algorithms if this situation persists for more than 15 seconds, and the result will often be that one or more hosts present errors to applications that result in application failures. Following these recommendations will hopefully avoid this problem. Note: This issue is not in any way specific to the SVC. All controllers and operating systems have the same issues if the maximum queue depth is reached.
84
Implementing the IBM System Storage SAN Volume Controller V4.3
Calculating a queue depth limit When calculating the queue depth, consider the following factors: Although the maximum number of queued commands is per node and there are two nodes in an I/O group, the system must continue to function when one of the nodes in an I/O group is not available. Thus, you must consider an I/O group to have the same number of queued commands as a node. However, when a node fails, the number of paths to each disk is halved. In practice, this effect can be neglected, and you can count nodes rather than I/O groups in the calculation below. If a VDisk is mapped so that it can be seen by more than one host, then each host that it is mapped to can send a number of commands to it. Multipathing drivers on some hosts round robin I/Os among the available I/O paths. For hosts that do not currently do this, it is possible that this behavior might change in the future, and you need to avoid “breaking” customers’ configurations when this happens. If a device driver times out a command, it will typically re-issue that command almost immediately. The SVC will have both the original command and the retry in the command queue in addition to any Error Recovery Process (ERP) commands that are issued. In order for the maximum queue depth not to be reached, then, for all VDisks associated with the I/O group, and for all hosts that are mapped to be able to see each VDisk, and for all paths on each host, the sum of the queue depths must be less than 10,000. Because ERPs can consume some number of queued command slots, this number is reduced to 7000 to allow a safety margin.
Homogeneous queue depth calculation This calculation applies to systems where: The available queued commands are to be shared out among all paths, rather than giving some hosts additional resources. The VDisks are shared out evenly among the I/O groups in the cluster.
Chapter 4. Performance and capacity planning
85
Note: The queue depth for each VDisk should be set on the hosts using the following calculation: q = Round up ((n * 7000) / (v * p*c)1 ) q = Per device path q-depth setting n = Number of nodes in the cluster v = Number of VDisks configured in the cluster p = Number of paths per VDisk per host. A path is a route from a host FC port to an SVC FC port that is recognized by the host as giving access to the VDisk. c= The number of hosts that can concurrently access each VDisk. Very few applications support concurrent access from multiple hosts to a single VDisk. Examples where multiple hosts have concurrent access to a disk include cases where the SAN File System (SFS) is in use. Thus, typically c will be 1. For example, we calculate the I/O queue for a homogenous SVC configuration with eight SVC nodes and the maximum number of supported VDisks. n=8: An 8 node SVC cluster v=4096: The number of VDisks per SVC cluster is a maximum of 4096. c=1: One host is able to access each VDisk. p=4: Each host has four paths to each VDisk (Two HBAs each with two paths to the I/O group). Calculate the queue depth as follows: roundup q= (8*7000) / (4096*4*1) = 4 So, the queue depth in the operating systems should be set to four concurrent commands per path.
Non-homogeneous queue depth calculation In some cases, it could be appropriate to give favored hosts additional resources to allow them to queue additional commands, or the number of VDisks supported by each I/O group might be different. In these cases, the queue depth is calculated in the following way. Note: Consider each I/O group in turn: For each VDisk, consider each host to which that VDisk has a mapping. This gives a set of (host and VDisk) pairs. As long as the total sum of all (host and VDisk) pairs (queue depth) is less than 7000, the system should not experience problems due to queue full situations. The above calculation assumes that there is a significant probability that all of the hosts will initiate the number of concurrent commands that they are limited to, that is, each host is busy. If there are a large number of fairly idle hosts in the configuration, which are not going to be initiating very many concurrent commands, then it reasonable that the queue depth does not need to be limited, even if the calculation above says that it should be. If this is the case, we recommend that the queue depth be increased/not set.
86
Implementing the IBM System Storage SAN Volume Controller V4.3
How to limit the queue depth Once you have determined the appropriate queue depth limit, you must apply it. Each operating system has an OS/HBA specific way to limit the queue depth on a per device path basis. An alternative to setting a per path limit is to set a limit on the HBA. Note: For example, one host has two HBAs and four paths per VDisk. The host will access 40 VDisks, and its maximum concurrent I/O commands per path is five. The calculation is: Four paths per VDisk, with a limit of five concurrent I/O commands per path, equals 20 concurrent I/O commands per VDisk. 40 VDisks with 20 concurrent I/O commands per VDisk equals 800 concurrent I/O commands for the host. Therefore, it may be appropriate to place a queue depth limit of (40*(4*5))/2=400 on each adapter. This allows sharing of the queue depth allocation between VDisks. For a system that is already configured, the result (v*p*c) is actually the same number, as determined by issuing the datapath query device command on all hosts, and summing up the number of paths.
4.1.4 Host Here are some considerations when installing a host into the SVC environment.
Balanced load across HBA ports To obtain the best performance from a host with multiple FC ports, the zoning should ensure that each FC port of a host is zoned with a different group of SVC ports (see Figure 4-1 on page 79).
Balanced host load across SVC ports To obtain the best performance of the subsystem and to prevent overloading, the workload to each SVC port should be equal. This will typically involve zoning approximately the same number of host FC ports to each SVC FC port.
Optimize multipath for SDD SDD takes advantage of redundant connections between a host server and the disk storage server to optimize availability and distribute I/O activity. It distributes the workload among the multiple I/O paths for load balancing. SDD uses either the load balancing (default) or round robin policy to select paths for I/O. The preferred paths will do load balancing. The best configuration we recommend is four paths per physical LUN on the host (two preferred paths and two backup paths). An example configuration that addresses this point is shown in Figure 4-1 on page 79. More than four paths may degrade the IO performance, while two paths are not reliable enough. In the case of one host or storage HBA, or if the switch malfunctions, there is only one good path left. If for some reason the storage microcode failed, or an I/O request timed out on the good path, then there is a possibility that the application may fail. In an environment where multiple servers are sharing LUNs, the total number of paths from all the servers to a LUN should be considered. Configuring more than four paths per LUN potentially increases the risk of reaching the maximum number of queued commands for the SVC cluster.
Chapter 4. Performance and capacity planning
87
4.2 Performance modeling and sizing At the pre-sales stage, you can use IBM Disk Magic™ tool to assist you in estimating how many SVC nodes can satisfy the performance requirement in a typical environment. At the post-sales stage, you can use the IBM Disk Magic tool to model an existing configuration with SVC already installed. Here is the link to the too; you will need to contact your IBM account representative for access to it: http://w3-03.ibm.com/sales/support/ShowDoc.wss?docid=Q947558L63209Z65&infotype=SK& infosubtype=S0&campaign=&lng=&node=brands%2CB5000&ftext=disk+magic
4.3 Performance monitoring In this section, we discuss some performance monitoring techniques.
4.3.1 Collecting performance statistics By default, performance statistics are not collected. You can start performance collection by using the svctask startstats command, as described in 9.12, “Listing dumps” on page 394, and you can stop them using the svctask stopstats command, as described in 9.2.8, “Stopping a statistics collection” on page 317. You can list the files using the lsiostatsdumps command. Statistics gathering is enabled or disabled on a cluster basis. When gathering is enabled, all nodes in the cluster gather statistics. SVC supports sampling periods of the gathering of statistics from 1 to 60 minutes in steps of one minute. The gathering of this data is coordinated at a cluster level. There are two sets of performance statistics: Cluster wide statistics Per-node statistics Important: Enabling statistics collection with an interval less than 15 minutes will only enable per-node statistics. Cluster wide statistics will not be collected at an interval less than 15 minutes.
4.3.2 Cluster wide statistics A number of statistics are collected for every Virtual Disk and every Managed Disk known to the cluster. The statistics reported are on a per-cluster rather than a per-node basis. Thus, for example, the count of I/Os for a given managed disk are the aggregate of the I/Os for that managed disk across all of the nodes in the cluster. At the end of each sampling period, the statistics gathered during the sampling period are written to files on the configuration node. Each sampling period results in the creation of one file for Virtual Disk statistics, and one file for Managed Disk statistics.
88
Implementing the IBM System Storage SAN Volume Controller V4.3
Statistics file naming The files generated are written to the directory /dumps/iostats. The file name is in the following format: m_stats_
__ for MDisk statistics. v_stats___ for VDisk statistics. The node_front_panel_id is the node from which the statistics are collected: The panel ID is taken from the current configuration node. The date is in the form YYMMDD. The time is in the form HHMMSS. Example 4-1 shows some typical MDisk and the VDisk statistics file names. Example 4-1 Filename of per cluster wide statistics IBM_2145:ITSOSVC42A:admin>svcinfo lsiostatsdumps id iostat_filename 12 m_stats_104603_070528_133722 13 v_stats_104603_070528_133722[email protected] :/dumps/iostats/* c:\temp\iostats The -load parameter is used to specify the session defined in PuTTY._ for MDisk statistics. Nv_stats_<node_front panel_id>__ for VDisk statistics. Nv_stats_<node_front panel_id>__ for Node statistics. The node_front_panel_id is the node from which the statistics are collected: The date is in the format of YYMMDD. The time is in the format of HHMMSS. Example 4-3 shows MDisk, VDisk, and the node statistics file names. Example 4-3 Node statistics filename IBM_2145:ITSO-CLS1:admin>svcinfo lsiostatsdumps id iostat_filename 0 Nm_stats_104603_070528_113203 1 Nv_stats_104603_070528_113203 2 Nn_stats_104603_070528_113203 3 Nm_stats_104603_070528_115707 4 Nv_stats_104603_070528_115707 5 Nn_stats_104603_070528_115707 6 Nm_stats_104603_070528_122211 7 Nv_stats_104603_070528_122211 8 Nn_stats_104603_070528_122211 9 Nm_stats_104603_070528_124714 10 Nv_stats_104603_070528_124714 11 Nn_stats_104603_070528_124714 12 Nn_stats_104603_070528_131218 on the AIX client type.[email protected] For all other countries, it is: mailto:[email protected] . Example 8-30 shows the VDisk Kanga0002 that we have allocated to our AIX server before we expand it. Here, the capacity is 5 GB, and the vdisk_UID is 60050768018301BF2800000000000016. Example 8-30 Expanding a VDisk on AIX. This gives this information for the Senegal_bas0001 before expanding the VDisk (Example 8-36 on page 241). Here we can see that the capacity is 10 GB, and also what the vdisk_UID is. To find what vpath this VDisk is on the Windows 2003 host, we use the SDD command, datapath query device, on the Windows host (Figure 8-13). Here we can see that the Serial 6005076801A180E9080000000000000F of Disk1 on the Windows host (Figure 8-13) matches the vdisk ID of “Senegal_bas0001” (Example 8-36 on page 241). To see the size of the volume on the Windows host, we use Disk Manager, as shown in Figure 8-13. Commands: /h | /help | -? | /? showcfg listvols add (separated by spaces) rem (separated by spaces) Configuration: set user Chapter 8. Host configuration trace [0-7] trustpassword <trustpassword> truststore <truststore location> usingSSL vssFreeInitiator <WWPN> vssReservedInitiator <WWPN> FlashCopyVer <1 | 2> (only applies to ESS) cimomPort cimomHost namespace targetSVC <svc_cluster_ip> backgroundCopy <0-100> Example 8-72 shows that the host Nile is logged into the SVC with two HBAs. Example 8-72 lshost Nile Where SCSI HBA The number of the SCSI HBA (may change). SCSI target The number of the SCSI target (may change). SCSI VDisk The number of the VDisk (never changes). disk partition The number of the disk partition (never changes). If the last number is not displayed, the name stands for the entire VDisk. command (Example 8-74). Example 8-74 Expanding a VDisk in SVC[email protected] :/tmp/[email protected] :/dumps/snap.104643.080617.002427.tgz c:\ snap.104643.080617.002427 | 597 kB | 597.7 kB/s | ETA: 00:00:00 | 100% 5. Upload the new software package using PuTTY Secure Copy. Enter the command as shown in Example 9-106. Example 9-106 pscp -load[email protected] :/home/admin/upgrade IBM2145_INSTALL_4.3.0.0-0 | 103079 kB | 9370.8 kB/s | ETA: 00:00:00 | 100% 6. Upload the SAN Volume Controller Software Upgrade Test Utility using PuTTY Secure Copy. Enter the command as shown in Example 9-107. Example 9-107 Upload utility[email protected] :/home/admin/upgrade IBM2145_INSTALL_svcupgrad | 11 kB | 12.0 kB/s | ETA: 00:00:00 | 100%[email protected] email_contact ITSO User email_contact_primary 555-1234 email_contact_alternate email_contact_location ITSO email_state running email_user_count 1 inventory_mail_interval 0 380[email protected] :/dumps/elogs/svcerrlog_100048_080624_170257 c:\temp\svcerrlog.txt svcerrlog.txt | 6390 kB | 3195.1 kB/s | ETA: 00:00:00 | 100% In order to use the Run option, you must know where your pscp.exe is located. In this case, it is in C:\Program Files\PuTTY\. This command copies the file called svcerrlog_100048_080624_170257 to the C:\temp directory on our local workstation and calls the file svcerrlog.txt. Open the file in WordPad (Notepad does not format the screen as well). You should see information similar to what is shown in Example 9-119. The list was truncated for the purposes of this example. Example 9-119 errlog in WordPad[email protected] :/dumps/feature/feature.txt c:\temp\feature.txt feature.txt | 18 kB | 18.5 kB/s | ETA: 00:00:00 | 100%[email protected] :/tmp/svc.config.backup.xml c:\temp\clibackup.xml clibackup.xml | 97 kB | 97.2 kB/s | ETA: 00:00:00 | 100% command. For intracluster: svctask chcluster -gmintradelaysimulation inter_cluster_delay_simulation and intra_cluster_delay_simulation express the amount of time (in milliseconds) secondary I/Os are delayed respectively for intercluster and intracluster relationships. These values specify the number of milliseconds that I/O activity, that is, copying a primary VDisk to a secondary VDisk, is delayed. A value from 0 to 100 milliseconds in 1 millisecond increments can be set for the cluster_delay_simulation in the commands above. A value of zero disables the feature. To check the current settings for the delay simulation, use the following command: svcinfo lscluster In Example 13-5, we show the modification of the delay simulation value and a change of the Global Mirror link tolerance parameters. We also show the changed values of the Global Mirror link tolerance and delay simulation parameters. Example 13-5 Delay simulation and link tolerance modification To list the number of free extents available on an MDisk, use the following CLI command: svcinfo lsfreeextents <mdiskname | mdisk_id> Important: After a migration has been started, there is no way for you to stop the migration. The migration runs to completion unless it is stopped or suspended by an error condition, or if the VDisk being migrated is deleted. and checking the fast_write_state attribute. This attribute has the following meanings: empty not_empty corrupt and checking the fast_write_state attribute. This attribute has the following meanings: empty and checking the fast_write_state attribute. This attribute has the following meanings: empty -a pv=clear chdev -l -a pv=yes Therefore, it is necessary to redefine the volume group information on the FlashCopy target using the recreatevg command. Refer to “AIX recreatevg command” on page 847 for further details. This alters the VGID of the FlashCopy target volume so that there are no conflicts with existing VGIDs on active volume groups. If you do not redefine the volume group information prior to importing the volume group, then the importvg command fails. 2. When using Subsystem Device Driver (SDD), the cfallvpath command should be run to discover all new vpath devices: cfallvpath Note: If you just execute cfgmgr or cfgmgr -vS, the SDD will discover the new disks and make the vpaths automatically. Sometimes the cfgmgr command can take a while and will not affect the server configuration at all, therefore the cfgmgr -l command can be the fastest way to discover the new disks. 3. Check which new MPIO volume or vpath is on the host; this is your FlashCopy target: lsdev -c disk | grep vpath (For vpath devices) lsdev -c disk | grep 2145 (For mpio devices) 6. The importvg command should vary on the volume group; if not, use the varyonvg command: varyonvg 7. Verify the access to the volume group using: lqueryvg -Atp 8. To list all file systems in the volume group, run: lsvg -l 9. Verify consistency of all file systems on the FlashCopy target: fsck -y 10.Mount all the target file systems: mount The data is now available. You could now perform a tape backup on this FlashCopy image. You can also use the data for other purposes, such as testing. This procedure can be run as soon as the relationship between the FlashCopy source and the target is established, even though data is still being copied from the source to the target in the background. It might be the case that the disks containing the target VDisks were previously defined to the target AIX system. For example, if you periodically do backups using the same VDisks, then in this case, perform the following actions on the target system before recreating the new image: 1. Unmount all file systems in the target volume group: umount 2. Vary off the volume group: varyoffvg 3. Export the volume group: exportvg 4. Delete the hdisks and vpaths or MPIO devices using the command rmdev: rmdev -Rdl dpo rmdev -dl Tip: rmdev -Rdl dpo will delete all the vpath devices. If there are too many hdisks, and you do not want to delete them one by one, you can also use rmdev -Rdl fcsx commands to delete all HBA devices and all child devices. 7. When using the Subsystem Device Driver (SDD), the cfallvpath command should be run to discover new vpath devices: cfallvpath 8. Check which new MPIO volume or vpath is on the host; this is your FlashCopy target: lsdev -c disk | grep vpath (For vpath devices) lsdev -c disk | grep 2145 (For mpio devices) 9. Verify that all the paths are working fine by running the following commands: datapath query device (For vpath device) pcmpath query device (For MPIO devices) 10.Import the volume group: importvg -y 11.The importvg command should vary on the volume group; if not, use the varyonvg command: varyonvg 12.Verify the access to the volume group using the following command: lqueryvg -Atp 13.To list all file systems in a volume group, run: lsvg -l 14.Verify consistency of all file systems on the FlashCopy target using the following command: fsck -y 15.Mount all the target file systems using the following command: mount The data is now available as a point in time recreation of the image. You could now perform a tape backup on this FlashCopy image or also use the data for other purposes, such as testing. or hd2vp rmdev -Rdl dpo rmdev -Rdl fcsx command takes a volume group and learns about possible changes performed to that volume group. Any new logical volumes created as a result of this command emulate the ownership, group identification, and permissions of the /dev special file for the volume group listed in the -y flag. The -L flag performs the functional equivalent of the -F and -n flags during execution. The following restrictions apply: The volume group must not be in an active state on the system executing the -L flag. The volume group’s disks must be unlocked on all systems that have the volume group varied on and operational. Volume groups and their disks might be unlocked, remain active, and used through the varyonvg -b -u command. The physical volume name provided must be of a good and known state; the disk named might not be in the missing or removed state. If an active node has both added and deleted logical volumes on the volume group, the -L flag might produce inconsistent results. The -L flag should be used after each addition or deletion, rather than being deferred until after a sequence of changes. If a logical volume name clash is detected, the command will fail. Unlike the basic importvg actions, clashing logical volume names will not be renamed. Here is an example of how to use the -L on a multi-tailed system: /F command on the target volume, to make Windows aware of the changes on the target volume. If you do not execute the chkdsk /F command, you cannot rely on the data. Using our previous example, the complete command would be chkdsk T: /F.[email protected] -i "C:\DirectoryPath\KeyName.PPK" 
