(Up to PS1 IPP Czar Logs)

Monday : 2012.11.05

• heather is czar (9:24). ipp missed her - hundreds of exposures stuck in registration... first attempt to unstick:

  src/ipptrunk-20120913/tools/regpeek.pl
  regtool -updateprocessedimfile -exp_id 544697 -class_id XY27 -set_state pending_burntool -dbname gpc1   
  

arrgh sticking again:

regtool -updateprocessedimfile -exp_id 544700 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544704 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544732 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544746 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544751 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544771 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544869 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544873 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544924 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544927 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544936 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544949 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544951 -class_id XY27 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 544970 -class_id XY27 -set_state pending_burntool -dbname gpc1

• 09:35 (Serge): Publishing nightlyscience made on 2012-11-02 and 2012-11-03 to IPP-MOPS_DEV (format version 3) for their tests.

• 10:45 (heather): gene rebooted ipp065, nfs was cranky - same machine as burntool failures...

• 11:30 (heather): gene says ipp012 is also nfs cranky - we have stopped all, waiting for queue to clear out, then gene will reboot 12 and heather will rebuild ippconfig (requested by chris)
• 21:15 (MEH): rebooting ipp033 for Heather Ipp033-crash-20121105
  • looks like summitcopy/registration pantasks got into odd state, going to restart them

Tuesday : 2012-11-06

Serge is czar. Heather is the Election czar.

• 06:30 (Serge): Still 89 exposures to download from the summit but things seem to run smoother than yesterday.
• 08:30 (Serge): All exposures successfully downloaded. A few 3PI still have to be processed.
• 09:45 (Serge): ns.add.date 2012-11-05
• 10:15 EAM : remove ipp023 - ipp026 from processing for final rsyncs
• 12:15 (Serge): Nightly processing for both previous nights complete

Wednesday : 2012-11-07

Bill is czar today

• 10:00 Summit copy has fallen behind by about 140 exposures, it appears that some jobs are getting stuck waiting for glockfile to complete for files on ipp009. I've set ipp009 to repair in nebulous.
• 10:26 restarted summit copy to clear the counts. We are getting a very high number of timeouts from -pendingfoo modes. Seems to be all tools.
• 10:50 shut down and power cycled ipp009 to clear up broken locks
• 10:54 restarted stdscience, stack, and distribution pantasks. Set update to stop.
• 11:10 ipp009 came up with no network. Gavin is investigating.

Thursday : 2012-11-08

Friday : 2012-11-09

Saturday : 2012-11-10

Sunday : 2012-11-11