PS1_IPP_Czarlog_20150608 – Pan-STARRS IPP

Context Navigation

Version 22 (modified by Mark Huber, 11 years ago) ( diff )
--

16:05 Bill: restarted pstamp pantasks. It was looking tired to me.

17:50 MEH: will start using the ippxNNN nodes for MD night stacks until the nodes are needed for other things

07:35 EAM : stdscience is way behind. I'm stopping and restarting it.
11:44 Bill: increased priority of label ps_ud_WEB to 500 (higher than nightly science) so that two chips can get processed for a MOPS postage stamp request. Once they are in the stdscience queue I will set it back.
15:00 CZW: Reduced shuffle from -j 24 to -j 12 to half the load.
16:30 CZW: Starting ipplanl/pv3shuffle pantasks which will execute neb-replicate commands for all the LANL stack products. These are targeted onto the stsciXY.Z nodes.
- 17:00 HAF restarting everything, will pull out the wave1s that are on the funky pdu - ipp004, ipp005, 006, 007, 009, 010, 011, 015, and ipp066 pulled out of pantasks for tonight.
22:20 EAM : removed ippc21, ippc55, ippc62 from processing lists -- not mounting home dirs?
22:20 EAM : i've killed off all hung jobs and umount the cab2 machines from all hosts. the cluster seems clear now. I'm going to restart ipp pantaskses.

05:36 Bill: changed label priority for ps_ud_MOPS to 500 which allow the small number (< 50 per day) of chip update jobs to be processed during nightly science processing.
07:42 Bill: added gpc2 database to summitcopy pantasks and reg.add.date for gpc2 to registration. That removed the gpc1 date from the registration date list. I guess the operation has changed
11:55 MEH: ipp066 has ganglia reporting down for 18+ hrs, just needs
```
sudo /etc/init.d/gmond restart
```
12:55 EAM: Haydn reports the cab2 machines are back up after he replaced the PDU. I've set them to neb repair.
12:59 Bill: restarted registration pantasks
13:36 Bill: dropped a repeatedly faulting warp: warptool -updateskyfile -set_quality 42 -fault 0 -warp_id 1589979 -skycell_id skycell.0857.004
15:20 MEH: with nightly processing ~finished, turning MD processing back on

15:45 MEH: gpc2 warp stuck -- cannot build growth curve (psf model is invalid everywhere)

warptool -dbname gpc2 -updateskyfile -set_quality 42 -skycell_id skycell.1207.015 -warp_id 7109 -fault 0

17:35 MEH: pstamp has been stopped for >2 hrs w/o reason? turning back to run for QUB stamps
20:40 MEH: cleanup wasn't disabled for MOPS data backlog from 20150610 and it got cleaned up...
- restart stdsci so fresh for this large amount of reprocessing.. -- also appears the ipp004,005,006,007,009,015 were still in a down state after their powerup
- LAP.PV3 warp_id 1159355 accidentally cleaned, will need to be updated w/ proper PV3 tag..
20:55 MEH: also removing MPE label from pstamp since QUB still doesn't have their stamps yet... -- put back in once QUB stamps finished
23:45 MEH: OSS update seems complete, turning off extra nodes in stdsci