PS1_IPP_Czarlog_20121022 – Pan-STARRS IPP

Context Navigation

Version 19 (modified by Mark Huber, 14 years ago) ( diff )
--

Mark is czar

00:00 MEH: after reboot ipp023, restarted summitcopy, registration, stdscience for clean start. ran regtool -revertprocessedimfile -dbname gpc1.
01:30 MEH: watching processing summit/registration getting stalled often, looks like ipp018 often not responding for replication stage, putting into repair (too many trying to replicate to ipp018 with so many red disks?). been a problem for a while with
```
INFO: task mount.nfs:29470 blocked for more than 120 seconds.
```
- seems to have now moved the problem to ipp017.. but handling better. ipp018 may want a reboot (been a whole ~23d since last one).
07:40 oddly, seems like manually running regtool was needed when this has been automatic in the past (and sometimes still is). has something changed in code related to it, is there a number of tries attempted and then stops?
```
regtool -updateprocessedimfile -exp_id 536377 -class_id XY11 -set_state pending_burntool -dbname gpc1
```
08:45 Bill restarted pstamp and update pantasks in order to have clear success and failure counts.
09:15 MEH: ipp023 down again.. same problem as last night. rebooting and taking out of processing Ipp023-crash-20121022
10:25 MEH: nightlyscience 99% through (finished downloading ~09:20)
12:15 MEH: Bill changed retention for M31+STS for MPG folks, when nightlyscience finished need to make install in src/ipp-20120802/ippconfig/recipes/ (shutting down and restarting all pantasks)