PS1 IPP Czar Logs for the week 2014.03.23 - 2014.03.29

(Up to PS1 IPP Czar Logs)

Monday : 2014.03.23

  • 03:04 Bill processing seems to be going smoothly. Checked the database and found a 537 long threadlist. There was one thread from ippc09 with a runtime of 5070 seconds. Killled it and the number of threads dropped to normal (~143). The previous troublesome thread (see last czarlog) was also from ippc09.
  • 05:37 Bill cleared repeating warp fault warptool -updateskyfile -set_quality 42 -fault 0 -warp_id 1515631 -skycell_id skycell.1474.065
  • 20:00 EAM purged ippdb00 binlogs to mysqld-bin.005022
  • 20:01 EAM stopping & restarting pv3diff pantasks

Tuesday : 2014.03.24

  • 08:24 Bill restarted postage stamp server pantasks. There are a lot of MPE requests waiting to be cleaned. I've removed the MPE label for now to allow that backlog to clear out.
    • 09:47 pstamp cleanup finished MPE label re-enabled
  • 16:17 CZW: Bill emailed that there were nebulous thread issues. I did a series of mysqladmin -u root kill 568262 commands, and this seems to have resolved the issue.
  • 19:55 EAM: stopping and restarting the pv3diff pantasks

Wednesday : 2014.03.25

  • 15:30 CZW: stopped processing so Haydn can change a battery on ipp094.
  • 16:30 CZW: shutdown/restarting ipp and pv3diff pantasks so that when we get the ipp094 all clear, they can start with fresh pcontrols.
  • 23:00 MEH: adding SNIa.WS.nightlyscience to regular processing now as well

Thursday : 2014.03.26

  • 03:50 Bill: 3 mysql threads connected to the nebulous database were stuck. Killed them off.

Friday : 2014.03.27

  • 04:?? Bill: killed off a stuck mysql thread. Got distracted and didn't check on processing progress and didn't notice ...
    • 05:17 There was a problem with burntool for one of the chips in o7108g0559o that blocked processing. Fixed the state and now we're catching up.
  • 06:05 Bill restarted postage stamp pantasks
  • 12:30 MEH: Haydn replaced ipp094 BBU the other day, it looks fine now and WB is enabled -- setting neb-host up
  • 19:20 EAM: restarting pv3diff and pv3diffleft. There have been a large number of errors due to double queued warps. there are not too many failures, I think, but they have been reverting and filling the queues. I am turning off diff.revert in both pantasks.

Saturday : 2014.03.28

  • 03:50 Bill: killed 3 stuck mysql threads. Processing isn't too far backed up. 40 incomplete downloads 17 pending chip runs. Where this really hurts the most is camera stage. It has so many files to look up and verify at the end that it backs up. 20 in progress currently
  • 04:48 Bill: stdscience pcontrol is at 100% cpu usage. fake stage is falling behind. Looks like stdscience should be restarted.
  • 18:39 Bill: restarted stdscience, registration, and summit copy pantasks

Sunday : 2014.03.29

  • 17:30 EAM: i am shutting down pantasks owned by ~ippsky so I can do a build of the new tag, which contains fixes to psastro (skycal) and psphot (fullforce).