(Up to PS1 IPP Czar Logs)

Monday : 2012.11.19

• reminder for czar: ipp020,ipp014 taken out of processing saturday from stalling jobs due to mounts. need more attention, likely a reboot during working hours (MEH)
• 09:50: Serge: Added SC.TEST.MOPS.PS1_DV3 to stdscience and publishing for mops tests
• 09:50 Serge: Queued some OSS WS diffs observed a fwe nights ago

  difftool -dbname gpc1 -definewarpstack -good_frac 0.2 -warp_label OSS.nightlyscience -data_group OSS.20121116
   -stack_label ecliptic.rp -set_label SC.TEST.MOPS.PS1_DV3 -set_data_group SC.TEST.MOPS.PS1_DV3.OSS 
  -set_workdir neb://@HOST@.0/sch/SC.TEST.MOPS.PS1_DV3.OSS -available -set_reduction SWEETSPOT_WS 
  -set_dist_group NODIST -rerun -simple
  

Tuesday : 2012.11.20

Bill is czar tomorrow

• 14:49 Bill restarted cleanup pantasks
• 15:50 Bill turned off chip cleanup in the cleanup pantasks. He is doing some testing of the new code in another pantasks. Will turn it back on at the end of the day.
• 16:07 Bill stopped processing because ipp014 and ipp020 are stuck
• 16:23 Bill rebooted ipp020 and ipp014 Restarted stdscience
• 20:47 Bill turned chip cleanup on, stopped testing

Wednesday : 2012.11.21

Bill is czar today

• 05:55 Set a number of our problem nodes to nebulous up all together. 16, 18, 8, 9, 10, 11, 12
• 08:00 Well that didn't work! Set all pantasks to stop in preparation for restart and host rebalancing for staticsky.
• 09:45 restarted stdscience. poll limit set to 80 for now. I'm not sure if the cluster has fully recovered from all of the wave 1 hangs
• 10:02 There are a number of engineering exposures that haven't registered yet. They all seem to be pending_burntool, but aren't getting run
• 11:16 deepstack is running LAP.ThreePi.20120706 staticsky on compute 3 nodes
• 11:48 Serge: Stopped replication from ippdb00 to ippc63. Started copy of ippc63/mysql to ippdb04... using ssh but we don't have any other way :( I expect it to be ompleted on Friday.
• 12:46 ipp014 paniced: BUG: unable to handle kernel paging request at ffffffff800073d4
• 14:43 Gene noticed ipp014 was wedged and sent an email to Bill who was at lunch
• 15:10 set pantasks to stop, power cycled ipp014
• 15:44 shut down stdscience. pkilled all of the ipp user's glockfile processes. restarted stdscience. set other pantasks to run
• 16:26 set LAP.ThreePi.20115% staticsky distribution runs to be cleaned

Thursday : 2012.11.22

• 09:53 Serge: rsync to ippdb04 complete. Restarted nebulous replication on ippc63.

Friday : YYYY.MM.DD

Bill is czar today

• 08:00 one of the nodes has been stuck for 60,000 seconds stopped all pantasks
• 08:25 Serge: Nebulous database is now being replicated to ippdb04.

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD