PS1 IPP Czar Logs for the week 2014.03.23 - 2014.03.29

(Up to PS1 IPP Czar Logs)

Monday : 2014.03.23

• 03:04 Bill processing seems to be going smoothly. Checked the database and found a 537 long threadlist. There was one thread from ippc09 with a runtime of 5070 seconds. Killled it and the number of threads dropped to normal (~143). The previous troublesome thread (see last czarlog) was also from ippc09.

• 05:37 Bill cleared repeating warp fault warptool -updateskyfile -set_quality 42 -fault 0 -warp_id 1515631 -skycell_id skycell.1474.065
• 20:00 EAM purged ippdb00 binlogs to mysqld-bin.005022
• 20:01 EAM stopping & restarting pv3diff pantasks

Tuesday : 2014.03.24

• 08:24 Bill restarted postage stamp server pantasks. There are a lot of MPE requests waiting to be cleaned. I've removed the MPE label for now to allow that backlog to clear out.
  • 09:47 pstamp cleanup finished MPE label re-enabled
• 16:17 CZW: Bill emailed that there were nebulous thread issues. I did a series of mysqladmin -u root kill 568262 commands, and this seems to have resolved the issue.
• 19:55 EAM: stopping and restarting the pv3diff pantasks

Wednesday : 2014.03.25

• 15:30 CZW: stopped processing so Haydn can change a battery on ipp094.
• 16:30 CZW: shutdown/restarting ipp and pv3diff pantasks so that when we get the ipp094 all clear, they can start with fresh pcontrols.
• 23:00 MEH: adding SNIa.WS.nightlyscience to regular processing now as well

Thursday : 2014.03.26

• 03:50 Bill: 3 mysql threads connected to the nebulous database were stuck. Killed them off.

Friday : 2014.03.27

• 04:?? Bill: killed off a stuck mysql thread. Got distracted and didn't check on processing progress and didn't notice ...
  • 05:17 There was a problem with burntool for one of the chips in o7108g0559o that blocked processing. Fixed the state and now we're catching up.
• 06:05 Bill restarted postage stamp pantasks

Saturday : 2014.03.28

Sunday : 2014.03.29