PS1_IPP_Czarlog_20150518 – Pan-STARRS IPP

Context Navigation

Version 13 (modified by Mark Huber, 11 years ago) ( diff )
--

16:23 CZW: restarting ippsky/pv3ff{lt|rt} because they appear to have been running forever.

05:45 EAM: stdscience is running somewhat slowly. i'm restarting it.
12:10 MEH: an abundance of ps_ud_QUB label diffs in goto_cleaned state not being cleaned again -- changing label to goto_cleaned to free space
20:00 EAM: restarting fforce
- 20:45 EAM: we are getting some excessively large fforce jobs on pv3ffrt from the Galactic plane. I'm removing storage nodes from that pantasks until we are past.

05:30 EAM: rebooting stsci03 -- processing is somewhat behind, but not yet clear if that is due to stsci03. (the reboot helped -- things are now moving along)
06:10 EAM: I'm restarting the pv3fflt,pv3ffrt pantasks. pv3ffrt is working on the plane -- it is slower than pv3fflt and it needs more memory per job. at the same time, pv3fflt is constantly running out of jobs because the queuing is barely able to keep up. I'm adjusting the loading to put all but 14 of the x nodes in pv3ffrt (since they have more memory), and leaving both storage and compute to pv3fflt (when stdscience does not need them) along with the 14 unused x nodes.
11:00 EAM: I allowed too many jobs to the x2 and x3a nodes for pv3ffrt : they are thrashing on too much memory. I am trying to kill off jobs and bring them down to 5 jobs instead of 9.

13:50 CZW: Started p1_clear.pl shuffle job on stsci15 volume 0, directory 00. This will add some disk IO, but should not overburden the host.
20:20 EAM: retarted stdscience and pv3ffrt.

09:00 MEH: more old distribution data to cleanup again
09:50 MEH: doing months of past bzip2 on the ipp pantasks logs.. symlink to export .1 Archive broken while on ippc19, but bzip should free up a good fraction
10:30 MEH: doing regular restart of nightly pantasks
13:00 MEH: apache neb servers low on disk space, best to cycle out the logs now rather than Friday before a long weekend..