PS1_IPP_Czarlog_20131209 – Pan-STARRS IPP

Context Navigation

Version 17 (modified by eugene, 13 years ago) ( diff )
--

17:50 EAM : light processing today (chris cleared an error-ed out stack). I restarted the gpc1 database and rebooted ipp007 to clear NFS problems with no trouble. I tried to do the same for stsci13 and it hung in the reboot. Haydn will power cycle tomorrow since we do not have console power access. stsci06 also needs to be rebooted, but I am going to wait until stsci13 is back up. I restarted processing with stsci13 and stsci06 in neb repair.

00:30 MEH: status of processing, 2 MD01 chips faulted and
```
240 summit exposures since 2013-12-10

82 incomplete downloads

102 exposures copied but not registered
```
- as logged in czarlog over weekend, ipp057, 063, 050 out of processing due to lock problems, got back in processing and lock problems ensued..
- looks like ipp031, 036 may be problematic as well, out they go..
- ipp031,032 may be behaving oddly, lost in nebulous disk use plot...
- czarpoll and roboczar were stopped on ippc11 -- restarted
- ipp015,011 as before as well. add ipp054 to the list out of processing
01:50 registration caught up to the bad weather.. processing moving w/o problems so far
Bill 11:28 Need to rerun SAS.20131205 skycal pantasks is running out of ~bills/sas.32. Should get finished pretty quickly

11:57 : stsci06 needs to be rebooted: not locking files. I'm stopping processing to reboot a number of machines: ipp031, ipp036, ipp050, ipp057, ipp063, stsci06
- ipp036 was running neb-replicate, but since file locking was failing, it probably is not getting far. it was still on --min 0
- ipp050 reports a failed battery on raid
- done with ipp031 : no problems, glock works again
- done with ipp036 : no problems, glock works again (12:31)