(Up to PS1 IPP Czar Logs)

Monday : 2012.11.05

• heather is czar (9:24). ipp missed her - hundreds of exposures stuck in registration... first attempt to unstick:

  src/ipptrunk-20120913/tools/regpeek.pl
  regtool -updateprocessedimfile -exp_id 544697 -class_id XY27 -set_state pending_burntool -dbname gpc1   
  

arrgh sticking again:

regtool -updateprocessedimfile -exp_id 544700 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544704 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544732 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544746 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544751 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544771 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544869 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544873 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544924 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544927 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544936 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544949 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544951 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544970 -class_id XY27 -set_state pending_burntool -dbname gpc1

• 09:35 (Serge): Publishing nightlyscience made on 2012-11-02 and 2012-11-03 to IPP-MOPS_DEV (format version 3) for their tests.

• 10:45 (heather): gene rebooted ipp065, nfs was cranky - same machine as burntool failures...

• 11:30 (heather): gene says ipp012 is also nfs cranky - we have stopped all, waiting for queue to clear out, then gene will reboot 12 and heather will rebuild ippconfig (requested by chris)
• 21:15 (MEH): rebooting ipp033 for Heather Ipp033-crash-20121105
  • looks like summitcopy/registration pantasks got into odd state, going to restart them

Tuesday : 2012-11-06

Serge is czar. Heather is the Election czar.

• 06:30 (Serge): Still 89 exposures to download from the summit but things seem to run smoother than yesterday.
• 08:30 (Serge): All exposures successfully downloaded. A few 3PI still have to be processed.
• 09:45 (Serge): ns.add.date 2012-11-05
• 10:15 EAM : remove ipp023 - ipp026 from processing for final rsyncs
• 12:15 (Serge): Nightly processing for both previous nights complete

Wednesday : 2012-11-07

Bill is czar today

• 10:00 Summit copy has fallen behind by about 140 exposures, it appears that some jobs are getting stuck waiting for glockfile to complete for files on ipp009. I've set ipp009 to repair in nebulous.
• 10:26 restarted summit copy to clear the counts. We are getting a very high number of timeouts from -pendingfoo modes. Seems to be all tools.
• 10:50 shut down and power cycled ipp009 to clear up broken locks
• 10:54 restarted stdscience, stack, and distribution pantasks. Set update to stop.
• 11:10 ipp009 came up with no network. Gavin is investigating.
• 11:35 Gavin swapped the ethernet interface on ipp009 and it is back online. Restarted czarpoll and roboczar.
• 11:53 restarted update pantasks but removed ps_ud_MOPS.2 so that MOPS doesn't compete with itself.
• 12:56 Now ipp012 is being a pest. Many jobs stuck waiting for glockffile to finish. 64 rpc.statd processes. Setting to repair in nebulous
• removed LAP and MD05.GR0.20121030 labels from stdscience. After nightly processing is done we need to shut everything down and check on the status of things.
• 13:30 Added LAP and MD05 labels back in to stdscience.
• 13:45 After removing ipp012 from nebulous no jobs remained stuck waiting for it. They eventually cleared out. There are 64 rpc.statd processes so I'm going to reboot it.
• 14:06 ipp012 is back up. Set to up in nebulous but off in stdscience for now.
• 14:41 ipp012 is blocking processing again. it's rpc.statd is running at 100% cpu. Set to repair in nebulous.
• 14:45 Set ThreePi camera and diff stage dist runs from 2010-2011 to be cleaned
• 15:30 MEH: restarting stdscience to setup new MD03.refstack, used tweak_ssdiff and WS+SSdiff running
• 18:34 summit copy is already behind. added another set of hosts
• 18:45 MEH: set summitcopy to stop, was overloading summit. not sure which were added, looked like wave3+4, so set those host off and turning summitcopy back on

Thursday : 2012-11-08

Friday : 2012-11-09

Saturday : 2012-11-10

Sunday : 2012-11-11