== PS1 IPP Czar Logs for the week 2015.01.19 - 2015.01.25 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.01.19 ===

 * 05:25 EAM: ipp062 crashed around 04:15, nothing on console.  processing completely stalled.  i'm rebooting it now.
 * 05:40 EAM: after the reboot of ipp062, processing back up and running.
 * 09:55 MEH: ipp084 is in WT state and may be dragging on processing, into repair
 * 10:20 MEH: two warps stalling OSS finish
{{{
warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1369542 -skycell_id skycell.1609.051
warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1369710 -skycell_id skycell.1609.051
}}}
 * 14:45 EAM: stopping & restarting stdlocal
 * 15:15 EAM: rebooting ippx016 -- load very high for no good reason, ganglia shows there has been no I/O since midnight -- I suspect an ethernet crash.
 * 15:20 EAM: oops: I just shutdown stdlocal accidentally.  restarting it again.

=== Tuesday : 2015.01.20 ===
 * 05:00 EAM: stdscience is running sluggishly; stopping it to restart it
 * 10:20 EAM: some warps with growth curve failures:
{{{
	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371670 -skycell_id skycell.1697.023
	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371761 -skycell_id skycell.1698.077
	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371765 -skycell_id skycell.1700.047
	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371802 -skycell_id skycell.1787.048
	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371947 -skycell_id skycell.1701.070
}}}
 * 12:00 EAM: ipp041 crashed, nothing on console. rebooting.
 * 12:20 EAM: ipp056 has hung mounts, trying to clear
 * 12:25 EAM: ipp056 all jammed up (kernel segfaults), rebooting.
 * 14:00 EAM: bad diff (growth curve, repeat failures):
{{{
difftool -dbname gpc1 -updatediffskyfile -fault 0 -set_quality 42 -diff_id 642344 -skycell_id skycell.1699.083
}}}
 * 15:08 CZW: starting up ~ipplanl/pv3stacksummary/ pantasks to process/reprocess PV3 stack summaries.  I'm going to run these on the s0-s3 nodes, with one instance of each.  The jobs are rather low-impact, with IO being the largest resource.  The server defaults to 10 running jobs, and I'll bump this up if the ipp6509-e 10G link stays clear.

=== Wednesday : 2015.01.21 ===
 * 06:50 EAM: ipp045 crashed, nothing on console. rebooting.
 * 09:25 EAM: stdscience was far behind due to ipp045.  I've added ippsNN and some ippxNN nodes (esp the x0b and x1b set -- on the n5k switch)

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===