PS1 IPP Czar Logs for the week 2013.11.11 - 2013.11.17

(Up to PS1 IPP Czar Logs)

Monday : 2013.11.11

• MEHL ipp052 has stalling jobs, looks like mysql could use a restart there -- taking out of all pantasks

Tuesday : 2013.11.12

• 13:41 Bill restarted distribution pantasks.

Wednesday : 2013.11.13

Thursday : 2013.11.14

Bill is czar today

• 9:00 restarted pantasks. Queued sts distribution bundles for cleanup. Started processing of STS.rp.2013.201107%
• 9:20 ipp065 is being overwhelmed by ipptopsps. Taking it out of stdscience host list and temporarily setting it to down in nebulous to avoid failures due to esentially inaccessible detrend files.
• 09:48 queued 28 mostly 3pi exposures of M31 for reprocessing.
• 10:02 ipp050 crashed. Power cycled it.
• 10:44 stdscience stopped. There are a vast number of M31 chip failures due to failure to read auxmask file problems.
• 11:08 found the problem. Actually 2 M31 labels removed.

Friday : 2013.11.15

bill is czar today

• queued some more sts data.
• 11:00 stopped stdscience and distribution for restart. There are several stdscience jobs that seem to be stuck.
• 11:30 ipp049 is having problems. Ganglia claims that ipp055 is down but it is not.
• 11:45 rebooted ipp049 (it's doing fsck)

Saturday : 2013.11.16

Sunday : 2013.11.17