PS1 IPP Czar Logs for the week 2014-01-06 - 2014-01-12

(Up to PS1 IPP Czar Logs)

Monday : 2014-01-06

Bill is czar today

• 10:15 stdscience has finished. Restarted postage stamp pantasks. Added 2 x compute3 and 2 x compute2
• 10:38 restarted cleanup pantasks adding label goto_cleaned_redo which is to rerun chip cleanup processing (need to implement label priorities...)
• 13:12 restarted pantasks except for cleanup which is busy and replication which is currently not in the list managed by check_system.sh It is working though
• 13:45 LAP label added into stdscience
• 13:55 big camera backlog for lap due to the troubles the other night. Increased pending count to 40 from nominal value 25. ( set.camera.poll 40 )
• 16:00 set camera poll back to 25

Tuesday : 2014-01-07

Bill is czar today

• 14:02 stopping processing in preparation for daily restart
  • 15:12 it took a long time for the running camera_exp processes to finish. It seems like the cluster may be bogged down as a side effect of running Chris' file check scripts.
  • Restarted pantasks expect for cleanup. Leaving LAP label enabled but it may not make much progress.
• 16:51 Chris made some changes and the sluggishness is much reduced perhaps gone. cleanup is still off.

Wednesday : 2014-01-08

• 11:04 Bill Camera stage seems to be the bottleneck in LAP processing. Increased camera poll limit from 25 to 50.

Thursday : 2014-01-09

Friday : 2014-01-10

Saturday : 2014-01-11

Sunday : 2014-01-12