PS1 IPP Czar Logs for the week 2013.04.2 2- 2013.04.28

(Up to PS1 IPP Czar Logs)

Monday : 2013.04.22

• 10:40 Bill: ippc21 is having autofs issues turned it off in stdscience. Gavin will reboot
• 10:47 Bill: stdscience is sluggish set to stop for restart
• 10:53 Bill: stdscience pantasks restarted. Camera set to off to let the camera runs that were finishing up in the previous pantasks finish.
• 11:08 camera.on
• 21:55 MEH: sending distribution MD08.GR0 stacks (year old) and MD05.GR0 stacks (6 months) to cleanup to recover some diskspace

Tuesday : 2013.04.23

• 06:05 Bill: distribution has a lot of data outstanding yet seems to be backed up. Restarted the pantasks. Increased poll limit from 200 to 400.
• 17:45 CZW: home directories are temporarily hard to get to while I accidentally DDOS them with some stacking tests that still have a testing flag enabled. I'm killing jobs, fixing that testing flag issue, and trying to restore service.
• 17:58 CZW: ippc18 is back to normal.

Wednesday : 2013.04.24

• 11:25 Bill: added 2 sets of compute2 and compute3 to pstamp pantasks since the cluster isn't too busy
• 13:25 Bill: removed compute3 hosts from pstamp pantasks and added them to update

Thursday : 2013.04.25

• 09:00 MEH: appears a large number of MD07.GR0 night stacks failed to be made. will need to be heavily using the system for updates and stacking when nightly processing is not running...
• 11:54 CZW: running MD09 deep stack test to check how much my ppStack changes impact that reduction. These are running on stare00-03. I expect it will take a day or so to finish, but if these need to be stopped before then, let me know.
• 16:55 MEH: stdscience really could use its regular restart, may help speed up the MD07 updates as well

Friday : 2013.04.26

• 9:05 lazy czar (heather) kicks registration using regpeek.pl - some exposures not yet registered.

Saturday : 2013.04.27

Sunday : 2013.04.28