PS1_IPP_Czarlog_20150601 – Pan-STARRS IPP

Context Navigation

Version 6 (modified by watersc1, 11 years ago) ( diff )
--

05:50 EAM : pv3fflt does not have enough work queued and pv3ffrt is running very slowly in the bulge, so I've moved all of the machines to pv3ffrt, with reduced numbers of jobs per machine to avoid memory problems. once we are past the bulge, even for the outer galactic plane, we can add more jobs to those machines.
12:00 CZW: power cycling stsci17.
12:05 EAM: I have restarted mysql replication on ippdb03 (from ippdb05). replication failed last week due to a corrupted binlog file. I rsynced the database yesterday to ippdb05's local disk, then rsynced it last night to ippdb03. this morning, I set up the replication configuration. After discussing it with Heather, we have decided to keep the smaller camera databases (isp, uip, ssp) and the test databases (megacam, ps2_tc3) out of the replication stream to ippdb03. Instead, I have rsynced those databases to ippc42 and will set them up to replicate there (we can conveniently use the same master log position information as for the other setup).

19:51 HAF: neb-host up on ipp065, 066,067, 068, 069, 070, 071, 072, 073, 074, 075, 076, 076, 077, 078, 079, 080, 081 since they have a lot of free space, we need some of it, and addstar diff needs to think about what it's done.
20:03 EAM: stopping pv3ffrt to bump up the number of machine connections. we are mostly past the depths of the bulge and can handle a bit more processing.

11:30 Moved 314 ff runs from .remote to .left label. They all failed on the last run on the UHcray, so I suspect they have some fault/quality issue that isn't correctly handled there.