PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2013.09.02

• 00:10 MEH: looks like registration is stuck on something, 36 behind
• 11:30 MEH: Serge reported a warp repeatedlt faulting, looks like no good sources (1 psf source) -- edge skycell so set quality 42 for now.. exposure publishing for MOPS now
• 11:40 MEH: LAP stalled, good time for regular restart stdsci and clearing all LAP chip faults so processing can resume..
• 12:10 MEH: MD01 warp qual held up stacks and SSdiff as well -- stacks finished and tweak_ssdiff to run
• 19:10 MEH: LAP near run out of chips again, ~50 to fix. more single chips only on ipp047 -- 5 more to the list for 17

  | neb://ipp047.0/gpc1/20111120/o5885g0116o/o5885g0116o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111125/o5890g0407o/o5890g0407o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0203o/o5891g0203o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0117o/o5891g0117o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0127o/o5891g0127o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0222o/o5891g0222o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111125/o5890g0419o/o5890g0419o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111120/o5885g0127o/o5885g0127o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111120/o5885g0190o/o5885g0190o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111125/o5890g0412o/o5890g0412o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111120/o5885g0175o/o5885g0175o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0226o/o5891g0226o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0228o/o5891g0228o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111125/o5890g0508o/o5890g0508o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111125/o5890g0496o/o5890g0496o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0260o/o5891g0260o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0257o/o5891g0257o.ota67.fits |
  

Tuesday : 2013.09.03

Bill is czar today

• 07:15 registration is stuck. Can't see why everybody has burntool_state = -14. Restarting pantasks.
• 07:25 Ah there it is. burntool was stuck because one chip had burntool_state = -1 and data_state = check_burntool. too early in the morning.
• forgot to mention, ipp028 claimed to be down in ganglia but was up. It looks like it restarted around 6am today. It is now showing up in ganglia
• 09:30 We got reports of slow access to the data store last night. This may have been related to the ipp028 outage, but I haven't investigated. For good measure though I have restarted apace on ippc17.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD