Index: /trunk/doc/design/hardware_mgt.tex
===================================================================
--- /trunk/doc/design/hardware_mgt.tex	(revision 2394)
+++ /trunk/doc/design/hardware_mgt.tex	(revision 2395)
@@ -64,5 +64,5 @@
 phase is broken down into three discrete stages.
 
-\subsubsection{Stage 1}
+\subsubsection{Setup}
 
 This covers the installation of the nodes, the network equipment plus specing
@@ -74,7 +74,8 @@
 
 The IPP nodes will arrive pre-installed into standard 42U racks along with 1-2
-additional racks for the core switch and management equipment.  The shipment
-will need to be unpackaged, inventoried, inspected for physical damage, and
-positioned in the data center.
+additional racks for the core switch and management equipment.  There will also
+be a small pool of hard disks included with the shipment for the testing of the
+RAID controllers.  The shipment will need to be unpackaged, inventoried,
+inspected for physical damage, and positioned in the data center.
 
 Once the racks are in their final configuration and the core switch has been
@@ -84,5 +85,5 @@
 shall be red PVC or Plenum rated Category 6 RJ-45 patch cables.
 
-Console server equipment and network managed power strips will then be ordered.
+Console server equipment and network managed power strips can then be ordered.
 The number and type of network managed power strips will depend on whether 110v
 or 220v electricity is available and the physical configuration of the cluster
@@ -115,5 +116,5 @@
 SystemImager\footnote{SystemImager - http://www.systemimager.org/} or another
 similar software image installation package.  All filesystems should use
-\code{ext3} with the obvious exception of of the swap space.
+\code{ext3} with the obvious exception of system swap space.
 
 The nodes shall each be assigned a permanent IP address in an RFC
@@ -123,11 +124,18 @@
 to be used for the IPP Cluster then special coordination will need to be made
 with the Pan-STARRS Global Systems group due to possible network routing issues
-between other PS1 data systems and the IPP Cluster.  The IP address allocation
-for the nodes shall be sequentially from the beginning of the address space
-with the first two IP address reserved for the gateway.  \tbd{The IP address of
-all of the IPP cluster nodes should be recorded into the} \file{/etc/hosts}
-file on every node to prevent DNS look-ups during inter-cluster communication.
+between the IPP Cluster and other PS1 data systems.  The IP address allocation
+for the nodes shall be sequential from the beginning of the address space with
+the first two IP address reserved for the gateway.  The IP address of all
+of the IPP cluster nodes should be recorded into the \file{/etc/hosts} file on
+every node to prevent DNS look-ups during inter-cluster communication.
 \tbd{Additional configuration based on some other document that describes IPP
 software requirements?}
+
+One of the nodes shall be designated as the centralized log host for Syslog
+and other monitor tools.  In particular, \code{3DM}\footnote{3DM -
+http://www.3ware.com/products/raid\_management.asp} shall be used for monitoring
+the health of the 3ware RAID controllers and attached disks.  \code{3DM} sends
+warnings and error notifications via SMTP.  The log host should be configured
+with a dedicated account to receives these messages.
 
 \begin{figure}
@@ -147,6 +155,6 @@
 \end{figure}
 
-The IPP Hardware Test Plan (\tbd{separate PSDC number?}) can then begin.  The
-testing will cover the following items.
+The IPP Hardware Test Plan should then be carried out.  The testing will cover
+at least the following items:
 
 \filbreak
@@ -182,16 +190,16 @@
 \end{itemize}
 
-\subsubsection{Stage 2}
+\subsubsection{Software Development}
 
 The cluster enters into a period of partial dormancy where all but 6 of the
 nodes are deactivated.  The remaining active nodes are to each have 12 disks
-from the pool of commissioning disks installed and configured as a single RAID 5
-volume mounted as \file{/export/[nodename]/vol0} and exported via NFS as
+from the pool of commissioning disks installed and configured as a single RAID
+5 volume mounted as \file{/export/[nodename]/vol0} and exported via NFS as
 \code{[nodename]}.  These nodes are to remain available for the testing of IPP
 software until the cluster enters into it's production phase.  The duration of
-this stage will vary as Stage 3 won't begging until PS1 is almost ready to
-start taking data.  The minimum duration of this stage is 2.5 months.
-
-\subsubsection{Stage 3}
+this stage will vary as Stage 3 won't begging until PS1 is almost ready for
+first light.  The minimum duration of this stage is 2.5 months.
+
+\subsubsection{Pre-Production}
 
 The $1^{st}$ lot of storage disks are ordered and the complete cluster is
@@ -218,5 +226,5 @@
 volume mounted as \file{/export/[nodename]/vol0} and shared via NFS by
 exporting \file{/export/[nodename]} as \code{[nodename]}.  The new disks shall
-then be run through the IPP Storage Test Plan.
+then be run through the ``Storage'' section of the IPP Hardware Test Plan.
 
 \begin{figure}
@@ -242,21 +250,23 @@
 
 Approximately 6 months after the IPP Cluster enters into the Production phase
-the $2^{nd}$ lot of storage disks will need to be ordered.  The new disks shall be
-installed in sets of 12 on a single RAID controller in nodes [ipp000..ipp049]
-and configured as a single RAID 5 volume mounted as
+the $2^{nd}$ lot of 600 storage disks will need to be ordered.  It is estimated
+that by this time the storage capacity will be at least 600B per disk.  The new
+disks shall be installed in sets of 12 on a single RAID controller in nodes
+[ipp000..ipp049] and configured as a single RAID 5 volume mounted as
 \file{/export/[nodename]/vol1} and exported via NFS as part of the original NFS
 sharing of \file{/export/[nodename]}.  The new lot of disks will also need to
-be run through the IPP Storage test plan.
+be run through the ``Storage'' section of the IPP Hardware Test Plan.
 
 \subsubsection{3rd Disk Lot}
 
 Approximately 14 months after the IPP Cluster enters into the Production phase
-the $3^{rd}$ lot of disks will need to be ordered.  The new disks shall be
+the $3^{rd}$ and final lot of 600 disks will need to be ordered.  Disk capacity
+should be at least 900GB per disk by this time.  The new disks shall be
 installed in sets of 24 on two RAID controllers in nodes [ipp050..ipp074] and
 configured as two RAID 5 volumes (one per controller) mounted as
-\file{/export/[nodename]/vol0} \& \file{/export/[nodename]/vol1}.  Both of these
-volumes will be shared via NFS by exporting \file{/export/[nodename]} as
-\code{[nodename]}.  This lot will also be run through the IPP Storage test
-plan.
+\file{/export/[nodename]/vol0} \& \file{/export/[nodename]/vol1}.  Both of
+these volumes will be shared via NFS by exporting \file{/export/[nodename]} as
+\code{[nodename]}.  This lot will also be run through the ``Storage'' section
+of the IPP Hardware Test Plan.
 
 \subsection{End of Life}
@@ -267,4 +277,30 @@
 deactivated.  The deactivated nodes shall be stored for possible reuse with
 PS4.  This phase will last approximate one year.
+
+\section{Maintenance}
+
+\subsection{Nodes}
+
+The IPP Cluster nodes, excluding data storage disks, will be covered under a 3
+parts and labor warranty by the OEM.  The significant costs to the Pan-STARRS
+project will be the labors costs of installation/removal of hardware
+components, the testing associated with the new hardware components, and
+possibly the expense of shipping parts back to the OEM.
+
+The storage disks, depending on the vendor each lot is purchased from, may or
+may not be under warranty from the OEM.  In the case that a disk warranty is
+not extended by the OEM, only disks that are covered under a warranty by the
+manufacturer will be acquired.  In such a case the Pan-STARRS project will be
+required to pay the return shipping expenses unless special terms can be
+negotiated directly with the disk manufacturer.
+
+\subsection{Network Equipment}
+
+The IPP Cluster core network switch will need to be placed under a
+SmartNet\footnote{SmartNet -
+http://www.cisco.com/en/US/products/svcs/ps3034/ps2827/ps2978/serv\_datasheet09186a0080092491.html}
+contract from Cisco.
+
+\tbd{The cost scheduled depends on the level of service contract.}
 
 \section{Schedule}
@@ -295,5 +331,5 @@
 \includegraphics[angle=-90]{pics/ps1_ipp_cluster_timeline.ps}
 }
-\caption{ \label{timeline} Timeline}
+\caption{ \label{timeline} Time-line}
 \end{center}
 \end{figure}