PS1 IPP Czar Logs for the week 2013.12.09 - 2013.12.15

(Up to PS1 IPP Czar Logs)

Monday : 2013.12.09

• 17:50 EAM : light processing today (chris cleared an error-ed out stack). I restarted the gpc1 database and rebooted ipp007 to clear NFS problems with no trouble. I tried to do the same for stsci13 and it hung in the reboot. Haydn will power cycle tomorrow since we do not have console power access. stsci06 also needs to be rebooted, but I am going to wait until stsci13 is back up. I restarted processing with stsci13 and stsci06 in neb repair.

Tuesday : 2013.12.10

• 00:30 MEH: status of processing, 2 MD01 chips faulted and

  240 summit exposures since 2013-12-10
  
  82 incomplete downloads
  
  102 exposures copied but not registered
  

  • as logged in czarlog over weekend, ipp057, 063, 050 out of processing due to lock problems, got back in processing and lock problems ensued..
  • looks like ipp031, 036 may be problematic as well, out they go..
  • ipp031,032 may be behaving oddly, lost in nebulous disk use plot...
  • czarpoll and roboczar were stopped on ippc11 -- restarted
  • ipp015,011 as before as well. add ipp054 to the list out of processing
• 01:50 registration caught up to the bad weather.. processing moving w/o problems so far
• Bill 11:28 Need to rerun SAS.20131205 skycal pantasks is running out of ~bills/sas.32. Should get finished pretty quickly

• 11:57 : stsci06 needs to be rebooted: not locking files. I'm stopping processing to reboot a number of machines: ipp031, ipp036, ipp050, ipp057, ipp063, ipp011, ipp015, stsci06
  • ipp036 was running neb-replicate, but since file locking was failing, it probably is not getting far. it was still on --min 0
  • ipp050 reports a failed battery on raid
  • done with ipp031 : no problems, glock works again
  • done with ipp036 : no problems, glock works again (12:31)
  • done with ipp050 : no problems, glock works again
  • done with ipp057 : no problems, glock works again (12:45)
  • done with ipp063 : no problems, glock works again (12:51)
  • done with ipp011 : required a power-cycle, glock works again
  • stsci06 hung on reboot (just like stsci13 yesterday). power cycle still does not work for me, so I sent a message to Gavin (13:17)
    • stsci06 rebooted by Gavin (14:07)
  • done with ipp015 : no problems, glock works again (14:20)
• 14:45 : EAM : launched rsync jobs to return stsci copies of ipp031 & ipp032 data to those machines, removing them from processing.

Wednesday : 2013.12.11

• no data processed last night

  Current Time 2013-12-11 18:52:14
  o6637g0671d  2013-12-11 17:06:42 01:45:32 ago
  
  673 summit exposures since 2013-12-11
  
  625 exposures copied but not registered
  
  first exposure:  o6637g0001d 685258
  last exposure:   o6637g0671d 685930
  

• 09:50 Rita rebooting ipp018 to swap drive -- ipp018 manually out of processing and neb-host down
  • leave out of processing and neb-repair to not detract from rebuilding
• 12:09 CZW: A bad exposure confused regtool. Added define.sunset macro in registration pantasks to allow registration to have dateobs_begin after this bad exposure to allow regtool to be not confused.
• 12:30 MEH will take over as czar since Heather on travel
• 13:10 MEH: WSdiff for 3PI have more moderate time crunch, remove ThreePi.WS.nightlyscience label until nightly finished and will tweak_ssdiff then as well.
  • LAP.ThreePi.20130717 label also out now stacks caught up

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD