PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2015.08.03

• 00:50 MEH: ganglia reporing many down systems and many faults happened -- someone start/turn something on recently...
• 01:55 MEH: pstamps for QUB don't seem to be loading/running.. try restarting pstamp.. otherwise have to look at in morning..
• 02:10 MEH: still reg and other faults w/ long running jobs in stdsci.. probably going to be backed up mess in morning
• 05:55 Bill: restarted pstamp pantasks after moving some broken requests out of the way
• 06:10 EAM: ganglia continue to report many machines down, but this was not true. In the suspicion that this was the result of using the xnodes for nightly processing, i have returned to the original loading for stdscience (compute and storage nodes). I stopped and restarted all ipp pantasks.
• 11:20 MEH: clearing couple repeat WS faults with quality 42

  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.2322.017 -diff_id 1186759  -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.2261.014 -diff_id 1186775  -fault 0
  

• 12:30 MEH: Haydn taking ipp082 down to fix BBU -- neb-host down in advance as normal.
  • 13:10 back up, will leave in repair until data to process and test neb-host up response
• 17:00 CZW: restarting cleanup pantasks.
• 23:45 MEH: gpc2 OSS.20150725 was cleaned before MOPS had chance to process... with poor weather so far, setting to update and test ipp082 in neb-host up
  • ipp082 responded as expected, should be able to stay neb-host up now

Tuesday : 2015.08.04

• 07:52 Bill: ippc30 is acting overloaded. I suspect that Johannes is being a bit too aggressive downloading his results. I'm removing the MPE label from the postage stamp processor for

a bit in order to allow it to settle down.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD