PS1_IPP_Czarlog_20140714 – Pan-STARRS IPP

Context Navigation

Version 13 (modified by watersc1, 12 years ago) ( diff )
--

mark is czar

camera on pump-down over weekend, should be back up tonight.
10:20 MEH: will start regular restart of pantasks in preparation for week -- looks like someone had started multiple pantasks_severs for summitcopy, registration, stdsicence, cleanup, stack. dummy one looks to have date July 10.
- looks like someone restarted pstamp and distribution this morning, but pstamp still has a dummy pantask_server running, so shutting both down and restarting..
- also need to rotate the apache logs and so will interrupt processing -- ippc02 has ~5G less than other machines on / and unclear why
10:50 MEH: distribution looks to be stopped, no note so turning back to run
16:00 MEH: summit not ready, no data tonight

15:45 CZW: Mark noticed that the new nebulous directories were causing write problems. The issue seems to be that ipp068 and ipp069 did not have their nebulous directories chowned properly, preventing the nebulous group from writing. I've chowned/chmoded these to match the other hosts. In addition, I've rsynced the /data/ippXXX.0/ipp/ directory contents (tess, tmp) onto ipp067-ipp070, so these match as well.
17:25 CZW: Started side pantasks by ipp user on ippc20 to regenerate missing stack summary products. These should finish before processing starts tomorrow night.

14:00 CZW: Haydn has moved ippdb01 to a (hopefully) better cooled location in the racks, and I've restarted the pantasks servers.

17:10 CZW: I've turned on s1, s2, s3 in stack pantasks so we can have some power devoted to stacks. I don't know what the correct full-rebalance solution is, but this should at least keep the jobs running.