PS1_IPP_Czarlog_20140106 – Pan-STARRS IPP

Context Navigation

Version 15 (modified by watersc1, 13 years ago) ( diff )
--

Bill is czar today

10:15 stdscience has finished. Restarted postage stamp pantasks. Added 2 x compute3 and 2 x compute2
10:38 restarted cleanup pantasks adding label goto_cleaned_redo which is to rerun chip cleanup processing (need to implement label priorities...)
13:12 restarted pantasks except for cleanup which is busy and replication which is currently not in the list managed by check_system.sh It is working though
13:45 LAP label added into stdscience
13:55 big camera backlog for lap due to the troubles the other night. Increased pending count to 40 from nominal value 25. ( set.camera.poll 40 )
16:00 set camera poll back to 25

Bill is czar today

14:02 stopping processing in preparation for daily restart
- 15:12 it took a long time for the running camera_exp processes to finish. It seems like the cluster may be bogged down as a side effect of running Chris' file check scripts.
- Restarted pantasks expect for cleanup. Leaving LAP label enabled but it may not make much progress.
16:51 Chris made some changes and the sluggishness is much reduced perhaps gone. cleanup is still off.

11:04 Bill Camera stage seems to be the bottleneck in LAP processing. Increased camera poll limit from 25 to 50.

CZW: 13:10 Just now noticed that czartool has been dead since the mhpcc communications interruption. Restarted, and now I'm going to work on clearing out LAP jobs that are stuck.

10:45 CZW: I noticed things were clogged earlier. I restarted apache on a number of the nebulous server hosts, which has allowed nebulous requests to complete again (neb-touch timed out, and a number of summit copy jobs had similar issues). This doesn't seem to have solved everything, and the load on stsci02 (250-260?) is very suspicious. I've asked Gavin to reboot this (although since it is accessible, we can do command line reboots. I did not realize this before now).

Note: See TracWiki for help on using the wiki.