PS1_IPP_Czarlog_20150601 – Pan-STARRS IPP

Context Navigation

Version 17 (modified by Mark Huber, 11 years ago) ( diff )
--

05:50 EAM : pv3fflt does not have enough work queued and pv3ffrt is running very slowly in the bulge, so I've moved all of the machines to pv3ffrt, with reduced numbers of jobs per machine to avoid memory problems. once we are past the bulge, even for the outer galactic plane, we can add more jobs to those machines.
12:00 CZW: power cycling stsci17.
12:05 EAM: I have restarted mysql replication on ippdb03 (from ippdb05). replication failed last week due to a corrupted binlog file. I rsynced the database yesterday to ippdb05's local disk, then rsynced it last night to ippdb03. this morning, I set up the replication configuration. After discussing it with Heather, we have decided to keep the smaller camera databases (isp, uip, ssp) and the test databases (megacam, ps2_tc3) out of the replication stream to ippdb03. Instead, I have rsynced those databases to ippc42 and will set them up to replicate there (we can conveniently use the same master log position information as for the other setup).

19:51 HAF: neb-host up on ipp065, 066,067, 068, 069, 070, 071, 072, 073, 074, 075, 076, 076, 077, 078, 079, 080, 081 since they have a lot of free space, we need some of it, and addstar diff needs to think about what it's done.
20:03 EAM: stopping pv3ffrt to bump up the number of machine connections. we are mostly past the depths of the bulge and can handle a bit more processing.

11:30 CZW: Moved 314 ff runs from .remote to .left label. They all failed on the last run on the UHcray, so I suspect they have some fault/quality issue that isn't correctly handled there.
15:10 CZW: Moved 1477 ff runs from .bulge to .remote that are dec > +30. These should be far enough away from the plane to not require special treatment. I'm queuing the remaining 20hr runs to the .left label, and towards the end of day today, I'll relabel non-faulted, not part of the 314 moved earlier runs to the .remote label as well, and relaunch the Cray processing on those. This should clear the "easy" parts of the sky out while the bulge consumes the local cluster.
17:50 CZW: Moving ff runs around. 5387 .left to .remote, in box: 300 < ra < 315, dec > 30. This moves the majority of the non-faulted, not 314 runs to the remote label to be queued and run on the UH cray. There are still 20hr runs being queued to the left queue. /data/ippc19.0/home/watersc1/ffUHcray.20150504/move/left2remote.cmds contains the command to move these to remote. Based on the area and number moved at the 15:10 move, I suspect the 19hr queue may need to be double checked to ensure that it ran up all the way to the edge of the galactic latitude cut.
10:50 HAF: registration jammed, reverts no thappening in regtool, manually did {{regtool -revertprocessedexp -fault 2 -dbname gpc1}} and sent email

07:35 MEH: cleared stalled warp -- cannot build growth curve (psf model is invalid everywhere)

warptool -dbname gpc1 -updateskyfile -set_quality 42 -skycell_id skycell.1829.005 -warp_id  1586420   -fault 0

08:30 MEH: doing regular restart of nightly pantasks
09:15 MEH: seems to be stalled diffs not fixed since the weekend -- and warps have been through auto-cleanup so now more work to fix..
10:35 MEH: roboczar sendmail configured for ippc18 which is down, so isn't working currently -- Gavin re-enabled sendmail on ippc18 and now we have nagios warning emails as well
- apparently plan was to swap back to ippc18 as primary home system today -- will be delayed to a later date
10:40 MEH: manually sending things to cleanup missed since Sunday with the skip of initday hour for auto-cleanup trigger..
14:15 CZW: running shuffle code on stsci16.[0-2] to test loading. This should not substantially impact processing.
14:30 MEH: Haydn needs to replace disk in ippx070 and requires reboot -- turning off in ~ippsky/pv3ffrt
15:40 MEH: Haydn finished and ippx070 up -- on again in ~ippsky/pv3ffrt