PS1 IPP Czar Logs for the week 2015.01.19 - 2015.01.25

(Up to PS1 IPP Czar Logs)

Monday : 2015.01.19

• 05:25 EAM: ipp062 crashed around 04:15, nothing on console. processing completely stalled. i'm rebooting it now.
• 05:40 EAM: after the reboot of ipp062, processing back up and running.
• 09:55 MEH: ipp084 is in WT state and may be dragging on processing, into repair
• 10:20 MEH: two warps stalling OSS finish

  warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1369542 -skycell_id skycell.1609.051
  warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1369710 -skycell_id skycell.1609.051
  

• 14:45 EAM: stopping & restarting stdlocal
• 15:15 EAM: rebooting ippx016 -- load very high for no good reason, ganglia shows there has been no I/O since midnight -- I suspect an ethernet crash.
• 15:20 EAM: oops: I just shutdown stdlocal accidentally. restarting it again.

Tuesday : 2015.01.20

• 05:00 EAM: stdscience is running sluggishly; stopping it to restart it
• 10:20 EAM: some warps with growth curve failures:

  	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371670 -skycell_id skycell.1697.023
  	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371761 -skycell_id skycell.1698.077
  	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371765 -skycell_id skycell.1700.047
  	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371802 -skycell_id skycell.1787.048
  	warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1371947 -skycell_id skycell.1701.070
  

• 12:00 EAM: ipp041 crashed, nothing on console. rebooting.
• 12:20 EAM: ipp056 has hung mounts, trying to clear
• 12:25 EAM: ipp056 all jammed up (kernel segfaults), rebooting.
• 14:00 EAM: bad diff (growth curve, repeat failures):

  difftool -dbname gpc1 -updatediffskyfile -fault 0 -set_quality 42 -diff_id 642344 -skycell_id skycell.1699.083
  

• 15:08 CZW: starting up ~ipplanl/pv3stacksummary/ pantasks to process/reprocess PV3 stack summaries. I'm going to run these on the s0-s3 nodes, with one instance of each. The jobs are rather low-impact, with IO being the largest resource. The server defaults to 10 running jobs, and I'll bump this up if the ipp6509-e 10G link stays clear.

Wednesday : 2015.01.21

• 06:50 EAM: ipp045 crashed, nothing on console. rebooting.
• 09:25 EAM: stdscience was far behind due to ipp045. I've added ippsNN and some ippxNN nodes (esp the x0b and x1b set -- on the n5k switch)

Thursday : 2015.01.22

• 14:10 EAM: all sort of work today: we stopped all processing and unmounted everything by ~12:30 so Haydn and Gavin could reconfig the network, adding an extra trunk link between the 6509e switch and ippcore. this was completed and I restarted stdlocal. unfortunately, my umount -a jobs also unmounted /export/ippxNN for all ippx hosts. that meant widespread warp failures until I could remount those. I'm going to restart stdlocal once the remaining jobs have cleared.
• 14:45 EAM: stdlocal is running, staticsky is now running as user ippsky, using m0, m1, x0b, x1b, x3 nodes (4x each).

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD