Index: /trunk/doc/design/hardware_mgt.tex =================================================================== --- /trunk/doc/design/hardware_mgt.tex (revision 2393) +++ /trunk/doc/design/hardware_mgt.tex (revision 2394) @@ -11,6 +11,6 @@ \project{Pan-STARRS Image Processing Pipeline} \organization{Institute for Astronomy} -\version{00} -\docnumber{PSDC-xxx-xxx} +\version{DR01} +\docnumber{PSDC-4xx-xxx} % note the use of the docnumber & version number: % the complete PSDC document number is given by @@ -23,5 +23,7 @@ \RevisionsStart % version Date Description -01 & 2004.10.28 & First draft \\ +DR00 & 2004.10.28 & First draft \\ +\hline +DR01 & 2004.11.21 & Second draft \\ \hline \RevisionsEnd @@ -72,7 +74,7 @@ The IPP nodes will arrive pre-installed into standard 42U racks along with 1-2 -additional racks for the core switch and management equipment. The racks will -need to be unpackaged, inspected for physical damage, and positioned in the -data center. +additional racks for the core switch and management equipment. The shipment +will need to be unpackaged, inventoried, inspected for physical damage, and +positioned in the data center. Once the racks are in their final configuration and the core switch has been @@ -82,38 +84,50 @@ shall be red PVC or Plenum rated Category 6 RJ-45 patch cables. -Console server server equipment and network managed power strips will then be -ordered. The number and type of network managed power strips will depend on -whether 110v or 220v electricity is available. The console server equipment and -managed power strips should inter-operate to provide a unified cluster -management system. +Console server equipment and network managed power strips will then be ordered. +The number and type of network managed power strips will depend on whether 110v +or 220v electricity is available and the physical configuration of the cluster +racks. The console server equipment and managed power strips selected +inter-operate to provide a unified cluster management system. +Cyclades\footnote{Cyclades - www.cyclades.com} has been identified as possible vendor for +this equipment. There will be a slack period of approximately one month awaiting order -fulfillment. +fulfillment of cabling and equipment. \paragraph{Step 2} -The network and console wiring will be installed and connected to the ether the -core switch or the console servers. The node naming convention shall be in the -format of "ippxxx", where "xxx" is an integer sequentially assigned from the -range [000..999]. Each node shall have one of it's gigabit Ethernet ports -connected to the core switch by a Category 6 network patch cable. The core -switch has four 48-port gigabit Ethernet modules and one fourth of the nodes -shall be connected to each of those modules. The core switch shall be -configured such that all of the Ethernet ports are in the same VLAN. - -The nodes shall have a \tbd{Linux 2.6 based operating system that meets the -requirements defined in Software Requirements Specification (PSDC-430-005)} -installed on them. This shall be accomplished by setting up temporary DHCP and -TFTP servers, booting the nodes from the network via PXE, and installing a -local operating system image with SystemImager. The nodes shall each be -assigned a permanent IP address in an RFC 1519 Class C equivalent or larger -address space. If private IP address space, as defined in RFC 1918, is to be -used for the IPP Cluster then special coordination will need to be made with -the Pan-STARRS Global Systems group due to possible network routing issues +The network and console wiring will be installed and connected to the core +switch, managed power strips, and the console servers. + +The cluster nodes will follow a standardize naming convention in the format of +"ippxxx", where "xxx" is an integer sequentially assigned from the range +[000..999]. Each node shall have one of it's gigabit Ethernet ports connected +to the core switch by a Category 6 network patch cable. The core switch has +four 48-port gigabit Ethernet modules and one fourth of the nodes shall be +connected to each of those modules. The core switch shall be configured such +that all of the Ethernet ports are in the same VLAN. + +The nodes shall have a Linux 2.6 based operating system that meets the +requirements defined in Software Requirements Specification\footnote{Software +Requirements Specification - PSDC-430-005}installed on them. This shall be +accomplished by setting up temporary DHCP and TFTP servers, booting the nodes +from the network via PXE, and installing a local operating system image with +SystemImager\footnote{SystemImager - http://www.systemimager.org/} or another +similar software image installation package. All filesystems should use +\code{ext3} with the obvious exception of of the swap space. + +The nodes shall each be assigned a permanent IP address in an RFC +1519\footnote{RFC 1519 - ftp://ftp.rfc-editor.org/in-notes/rfc1519.txt} Class C +equivalent or larger address space. If private IP address space, as defined in +RFC 1918\footnote{RFC 1918 - ftp://ftp.rfc-editor.org/in-notes/rfc1918.txt}, is +to be used for the IPP Cluster then special coordination will need to be made +with the Pan-STARRS Global Systems group due to possible network routing issues between other PS1 data systems and the IPP Cluster. The IP address allocation for the nodes shall be sequentially from the beginning of the address space with the first two IP address reserved for the gateway. \tbd{The IP address of -all of the IPP cluster nodes should be recorded into the "/etc/hosts" file on -every node to prevent DNS look-ups during inter-cluster communication} +all of the IPP cluster nodes should be recorded into the} \file{/etc/hosts} +file on every node to prevent DNS look-ups during inter-cluster communication. +\tbd{Additional configuration based on some other document that describes IPP +software requirements?} \begin{figure} @@ -133,5 +147,38 @@ \end{figure} -The IPP Cluster Test Plan can then begin. +The IPP Hardware Test Plan (\tbd{separate PSDC number?}) can then begin. The +testing will cover the following items. + +\filbreak +Nodes: + +\begin{itemize} +\item +Power-supplies +\item +Memory +\item +CPUs +\item +PCI device identification +\item +Boot disk +\item +Network interfaces +\item +RAID controllers +\end{itemize} + +\filbreak +Core Switch: + +\begin{itemize} +\item +Memory +\item +Modules +\item +Hardware Diagnostics +\end{itemize} \subsubsection{Stage 2} @@ -139,14 +186,15 @@ The cluster enters into a period of partial dormancy where all but 6 of the nodes are deactivated. The remaining active nodes are to each have 12 disks -from the pool of commisioning disks installed and configured as a single RAID 5 -volume. These nodes are to remain available for the testing of IPP software -until the cluster enters into it's production phase. The duration of this -stage will vary as Stage 3 won't begging until PS1 is almost ready to start -taking data. The minimum duration of this stage is 2.5 months. +from the pool of commissioning disks installed and configured as a single RAID 5 +volume mounted as \file{/export/[nodename]/vol0} and exported via NFS as +\code{[nodename]}. These nodes are to remain available for the testing of IPP +software until the cluster enters into it's production phase. The duration of +this stage will vary as Stage 3 won't begging until PS1 is almost ready to +start taking data. The minimum duration of this stage is 2.5 months. \subsubsection{Stage 3} -The $1^{st}$ lot of storage disks are ordered and the complete cluster is brought -on line and prepared for production. +The $1^{st}$ lot of storage disks are ordered and the complete cluster is +brought on line and prepared for production. \paragraph{Step 1} @@ -154,18 +202,20 @@ Approximately 4 months before the IPP Cluster needs to be fully operational the first of three lots of 600 disks shall be ordered. The disks ordered should be -of the highest capacity commonly avaiable. It is estimated that this will be -400GB per disk when this stage is reached. The disks maybe purchased loose or -pre-installed in drive sleds if possible. The disks must have a 3 year -warrenty either from the OEM or the disk manufacturer. There will be a slack -period of approximately two months while the purchase is approved and delivery -is completed. +of the highest capacity commonly available. It is estimated that this will be +at least 400GB per disk when this stage is reached. The disks maybe purchased +loose or pre-installed in drive sleds if possible. The disks must have a 3 +year warranty either from the OEM or the disk manufacturer. There will be a +slack period of approximately two months while the purchase is approved and +delivery is completed. \paragraph{Step 2} -All IPP nodes are activated and subsections x-y of the IPP Cluster Test Plan -are re-run. If the $1^{st}$ lot of disks are purcashed lose they will be -concurrently mounted into drive sleds, the disk mounting should take -approximately 50 hours of labor. The new disks should be installed in sets of -12 on a single RAID controller in nodes [ipp000..ipp049]. The new disks shall +All IPP nodes are activated and subsections x-y of the IPP Hardware Test Plan +are re-run. If the $1^{st}$ lot of disks are purchased lose they will be +concurrently mounted into drive sleds, sled mounting should take approximately +50 hours of labor. The new disks should be installed in sets of 12 on a single +RAID controller in nodes [ipp000..ipp049] and configured as a single RAID 5 +volume mounted as \file{/export/[nodename]/vol0} and shared via NFS by +exporting \file{/export/[nodename]} as \code{[nodename]}. The new disks shall then be run through the IPP Storage Test Plan. @@ -184,5 +234,5 @@ includes replacing failed or about to fail disks, memory and other hardware components. The cluster will need to be checked daily for disks in need of -replacement. Occasionaly, new operating system images and software will need +replacement. Occasionally, new operating system images and software will need to be installed on the cluster. Additionally, there will be two new lots of disks purchased in this phase that will be passed through the IPP Storage Test @@ -192,14 +242,21 @@ Approximately 6 months after the IPP Cluster enters into the Production phase -$2^{nd}$ lot of storage disks will need to be ordered. The new disks shall be -installed in sets of 12 on a single RAID controller in nodes [ipp000..ipp049]. -The new lot of disks will also need to be run through the IPP Stoage test plan. +the $2^{nd}$ lot of storage disks will need to be ordered. The new disks shall be +installed in sets of 12 on a single RAID controller in nodes [ipp000..ipp049] +and configured as a single RAID 5 volume mounted as +\file{/export/[nodename]/vol1} and exported via NFS as part of the original NFS +sharing of \file{/export/[nodename]}. The new lot of disks will also need to +be run through the IPP Storage test plan. \subsubsection{3rd Disk Lot} Approximately 14 months after the IPP Cluster enters into the Production phase -the 3rd lot of disks will need to be ordered. The new disks shall be installed -in sets of 24 on two RAID controllers in nodes [ipp050..ipp074]. This lot will -also be run through the IPP Storage test plan. +the $3^{rd}$ lot of disks will need to be ordered. The new disks shall be +installed in sets of 24 on two RAID controllers in nodes [ipp050..ipp074] and +configured as two RAID 5 volumes (one per controller) mounted as +\file{/export/[nodename]/vol0} \& \file{/export/[nodename]/vol1}. Both of these +volumes will be shared via NFS by exporting \file{/export/[nodename]} as +\code{[nodename]}. This lot will also be run through the IPP Storage test +plan. \subsection{End of Life} @@ -242,361 +299,3 @@ \end{figure} - -\clearpage - -\appendix - -\section{IPP Cluster Test Plan} - -\subsection{Scope} - -\subsubsection{Identification} - -The Image Processing Pipeline Cluster hardware, including the processing nodes -and core network switch. - -\subsubsection{System overview} - -Receives, processes, and storages image data generated by the PS1 telescope -system. - -\subsubsection{Document overview} - -This document describes the commissioning test plan for the IPP Cluster -hardware. - -\subsection{Software test environment} - -\subsubsection{MHPCC} - -\subsubsection{Software} - -\subsubsection{Hardware} - -\subsubsection{Installation, testing, and control} - -\subsubsection{Personnel} - -2 FTEs. - -\subsubsection{Planned tests} - -\paragraph{Node} - -\subparagraph{physical inventory} - -\subparagraph{power supplies} - -All 3 power supplies have the green status led lit and have spinning fans. - -\subparagraph{Physical memory} - -Memory passes a complete set of memtest86+ and runs memtest86+ for atleast an -hour. - -\subparagraph{O/S boots} - -The system can boot from a Linux 2.6 based bootable cdrom without having a -kernel panic. - -\subparagraph{CPUs} - -`cat /proc/cpuinfo` show both CPUs at the proper clockrate. - -\begin{verbatim} -processor : 0 -vendor_id : AuthenticAMD -cpu family : 15 -model : 5 -model name : AMD Opteron(tm) Processor 248 -stepping : 10 -cpu MHz : 2189.901 -cache size : 1024 KB -fpu : yes -fpu_exception : yes -cpuid level : 1 -wp : yes -flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow -bogomips : 4325.37 -TLB size : 1088 4K pages -clflush size : 64 -cache_alignment : 64 -address sizes : 40 bits physical, 48 bits virtual -power management: ts fid vid ttp - -processor : 1 -vendor_id : AuthenticAMD -cpu family : 15 -model : 5 -model name : AMD Opteron(tm) Processor 248 -stepping : 10 -cpu MHz : 2189.901 -cache size : 1024 KB -fpu : yes -fpu_exception : yes -cpuid level : 1 -wp : yes -flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow -bogomips : 4374.52 -TLB size : 1088 4K pages -clflush size : 64 -cache_alignment : 64 -address sizes : 40 bits physical, 48 bits virtual -power management: ts fid vid ttp -\end{verbatim} - -\subparagraph{Memory} - -`cat /proc/meminfo` shows the full ammount of memory. - -\begin{verbatim} -MemTotal: 4026236 kB -MemFree: 3140404 kB -Buffers: 277248 kB -Cached: 55476 kB -SwapCached: 0 kB -Active: 301168 kB -Inactive: 34536 kB -HighTotal: 0 kB -HighFree: 0 kB -LowTotal: 4026236 kB -LowFree: 3140404 kB -SwapTotal: 16008764 kB -SwapFree: 16008764 kB -Dirty: 16 kB -Writeback: 0 kB -Mapped: 5576 kB -Slab: 532316 kB -Committed_AS: 9380 kB -PageTables: 448 kB -VmallocTotal: 536870911 kB -VmallocUsed: 8672 kB -VmallocChunk: 536862139 kB -HugePages_Total: 0 -HugePages_Free: 0 -Hugepagesize: 2048 kB -\end{verbatim} - -\subparagraph{PCI} - -`lspci` identifies all of the appropriate hardware. - -\begin{verbatim} -0000:00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) -0000:00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) -0000:00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) -0000:00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02) -0000:00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05) -0000:00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) -0000:00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) -0000:00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) -0000:00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) -0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration -0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map -0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller -0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control -0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration -0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map -0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller -0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control -0000:01:03.0 RAID bus controller: 3ware Inc 3ware Inc 3ware 9xxx-series SATA-RAID -0000:02:03.0 RAID bus controller: 3ware Inc 3ware Inc 3ware 9xxx-series SATA-RAID -0000:02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) -0000:02:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) -0000:03:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) -0000:03:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) -0000:03:05.0 Unknown mass storage controller: Silicon Image, Inc. (formerly CMD Technology Inc) SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02) -0000:03:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) -0000:03:08.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 10) -\end{verbatim} - -\subparagraph{Internal Harddisk} - -`fdisk -l` shows the internal hard disk - -\begin{verbatim} -Disk /dev/sda: 250.0 GB, 250059350016 bytes -255 heads, 63 sectors/track, 30401 cylinders -Units = cylinders of 16065 * 512 = 8225280 bytes - - Device Boot Start End Blocks Id System -/dev/sda1 1 13 104391 83 Linux -/dev/sda2 14 2006 16008772+ 82 Linux swap / Solaris -/dev/sda3 2007 3999 16008772+ 83 Linux -/dev/sda4 4000 30401 212074065 83 Linux -\end{verbatim} - -\subparagraph{Network} - -both gige interfaces can be configured and a test device pinged. - -\subparagraph{RAID controllers} - -`tw\_cli` shows both RAID controllers and reports there status as OK with no -alarms reported. - -\begin{verbatim} -//alae> alarms - -Ctl Date Severity Alarm Message ------------------------------------------------------------------------------- - -//alae> info c0 u0 - -Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks ------------------------------------------------------------------------ -u0 RAID-5 OK - - 64K 1629.74 3417817088 -u0-0 DISK OK - p0 - 232.82 488259584 -u0-1 DISK OK - p1 - 232.82 488259584 -u0-2 DISK OK - p2 - 232.82 488259584 -u0-3 DISK OK - p3 - 232.82 488259584 -u0-4 DISK OK - p4 - 232.82 488259584 -u0-5 DISK OK - p5 - 232.82 488259584 -u0-6 DISK OK - p6 - 232.82 488259584 -u0-7 DISK OK - p7 - 232.82 488259584 - -//alae> info c1 u0 - -Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks ------------------------------------------------------------------------ -u0 RAID-5 OK - - 64K 1629.74 3417817088 -u0-0 DISK OK - p0 - 232.82 488259584 -u0-1 DISK OK - p1 - 232.82 488259584 -u0-2 DISK OK - p2 - 232.82 488259584 -u0-3 DISK OK - p3 - 232.82 488259584 -u0-4 DISK OK - p4 - 232.82 488259584 -u0-5 DISK OK - p5 - 232.82 488259584 -u0-6 DISK OK - p6 - 232.82 488259584 -u0-7 DISK OK - p7 - 232.82 488259584 -\end{verbatim} - -\paragraph{Network hardware} - -\subparagraph{physical inventory} - -\subparagraph{Memory} - -`sh version` reports the proper ammount of memory. - -\begin{verbatim} -#sh ver -Cisco Internetwork Operating System Software -IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-I9K2S-M), Version 12.1(20)EW1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) -TAC Support: http://www.cisco.com/tac -Copyright (c) 1986-2003 by cisco Systems, Inc. -Compiled Mon 01-Dec-03 19:45 by eaarmas -Image text-base: 0x00000000, data-base: 0x0106BC90 - -ROM: 12.1(12r)EW -Dagobah Revision 86, Swamp Revision 28 - -goat uptime is 43 weeks, 1 day, 5 hours, 42 minutes -System returned to ROM by reload -System restarted at 12:12:06 US/Hawaii Sun Jan 4 2004 -System image file is "bootflash:cat4000-i9k2s-mz.121-20.EW1.bin" - - -This product contains cryptographic features and is subject to United -States and local country laws governing import, export, transfer and -use. Delivery of Cisco cryptographic products does not imply -third-party authority to import, export, distribute or use encryption. -Importers, exporters, distributors and users are responsible for -compliance with U.S. and local country laws. By using this product you -agree to comply with applicable laws and regulations. If you are unable -to comply with U.S. and local laws, return this product immediately. - -A summary of U.S. laws governing Cisco cryptographic products may be found at: -http://www.cisco.com/wwl/export/crypto/tool/stqrg.html - -If you require further assistance please contact us by sending email to -export@cisco.com. - -cisco WS-C4506 (XPC8245) processor (revision 7) with 524288K bytes of memory. -Processor board ID FOX073501MS -Last reset from Reload -5 Ethernet/IEEE 802.3 interface(s) -1 FastEthernet/IEEE 802.3 interface(s) -242 Gigabit Ethernet/IEEE 802.3 interface(s) -403K bytes of non-volatile configuration memory. - -Configuration register is 0x2102 -\end{verbatim} - -\subparagraph{Modules} - -`sh module` shows all the modules with status OK - -\begin{verbatim} -Chassis Type : WS-C4506 - -Power consumed by backplane : 10 Watts - -Mod Ports Card Type Model Serial No. ----+-----+--------------------------------------+------------------+----------- - 1 2 1000BaseX (GBIC) Supervisor(active) WS-X4515 JAE07400KBU - 2 48 10/100/1000BaseTX (RJ45) WS-X4448-GB-RJ45 JAB073904TB - 3 48 10/100/1000BaseTX (RJ45) WS-X4448-GB-RJ45 JAB073904M5 - 4 48 10/100/1000BaseTX (RJ45) WS-X4448-GB-RJ45 JAB06170832 - 5 48 10/100/1000BaseTX (RJ45) WS-X4448-GB-RJ45 JAB0617083X - 6 48 10/100/1000BaseTX (RJ45)V WS-X4548-GB-RJ45V JAB082109AX - - M MAC addresses Hw Fw Sw Status ---+--------------------------------+---+------------+----------------+--------- - 1 000d.ed4a.ac80 to 000d.ed4a.ac81 3.0 12.1(12r)EW 12.1(20)EW1, EAR Ok - 2 000d.bc8d.8360 to 000d.bc8d.838f 1.1 Ok - 3 000d.bc8d.7b50 to 000d.bc8d.7b7f 1.1 Ok - 4 0005.5fa4.38f0 to 0005.5fa4.391f 1.0 Ok - 5 0005.5fa4.3950 to 0005.5fa4.397f 1.0 Ok - 6 0011.20ea.bb00 to 0011.20ea.bb2f 2.0 Ok -\end{verbatim} - -\subparagraph{Diagnostics} - -`sh diag power-on` - -\begin{verbatim} - -Power-On-Self-Test Results for ACTIVE Supervisor - - -Power-on-self-test for Module 1: WS-X4515 - Port/Test Status: (. = Pass, F = Fail) - Reset Reason: Software/User - - -Port Traffic: L2 Serdes Loopback ... - 0: . 1: . 2: . 3: . 4: . 5: . 6: . 7: . 8: . 9: . 10: . 11: . -12: . 13: . 14: . 15: . 16: . 17: . 18: . 19: . 20: . 21: . 22: . 23: . -24: . 25: . 26: . 27: . 28: . 29: . 30: . 31: . - - -Port Traffic: L2 Asic Loopback ... - 0: . 1: . 2: . 3: . 4: . 5: . 6: . 7: . 8: . 9: . 10: . 11: . -12: . 13: . 14: . 15: . 16: . 17: . 18: . 19: . 20: . 21: . 22: . 23: . -24: . 25: . 26: . 27: . 28: . 29: . 30: . 31: . - - -Port Traffic: L3 Asic Loopback ... - 0: . 1: . 2: . 3: . 4: . 5: . 6: . 7: . 8: . 9: . 10: . 11: . -12: . 13: . 14: . 15: . 16: . 17: . 18: . 19: . 20: . 21: . 22: . 23: . -24: . 25: . 26: . 27: . 28: . 29: . 30: . 31: . au: . - - -Switch Subsystem Memory ... - 1: . 2: . 3: . 4: . 5: . 6: . 7: . 8: . 9: . 10: . 11: . 12: . -13: . 14: . 15: . 16: . 17: . 18: . 19: . 20: . 21: . 22: . 23: . 24: . -25: . 26: . 27: . 28: . 29: . 30: . 31: . 32: . 33: . 34: . 35: . 36: . -37: . 38: . 39: . 40: . 41: . 42: . 43: . 44: . 45: . 46: . 47: . 48: . -49: . 50: . 51: . 52: . 53: . 54: . 55: . - - -Module 1 Passed -\end{verbatim} - -\paragraph{Management hardware} - -\pagebreak -\section{IPP Storage Test Plan} - \end{document} -