PS1 IPP Czar Logs for the week 2010.09.02 - 2010.09.08

(Up to PS1 IPP Czar Logs)

Monday : 2013.09.02

• 00:10 MEH: looks like registration is stuck on something, 36 behind
• 11:30 MEH: Serge reported a warp continues faulting, looks like no good sources (1 psf source) -- edge skycell so set quality 42 for now.. exposure publishing for MOPS now

  warptool -dbname gpc1 -updateskyfile -warp_id 836928 -skycell_id skycell.049 -fault 0 -set_quality 42
  

• 11:40 MEH: LAP stalled, good time for regular restart stdsci and clearing all LAP chip faults so processing can resume..
• 12:10 MEH: MD01 warp qual held up stacks and SSdiff as well -- stacks finished and tweak_ssdiff to run
• 19:10 MEH: LAP near run out of chips again, ~50 to fix. more single chips only on ipp047 -- 5 more to the list for 17

  | neb://ipp047.0/gpc1/20111120/o5885g0116o/o5885g0116o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111125/o5890g0407o/o5890g0407o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0203o/o5891g0203o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0117o/o5891g0117o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0127o/o5891g0127o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0222o/o5891g0222o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111125/o5890g0419o/o5890g0419o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111120/o5885g0127o/o5885g0127o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111120/o5885g0190o/o5885g0190o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111125/o5890g0412o/o5890g0412o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111120/o5885g0175o/o5885g0175o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0226o/o5891g0226o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0228o/o5891g0228o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111125/o5890g0508o/o5890g0508o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111125/o5890g0496o/o5890g0496o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0260o/o5891g0260o.ota67.fits | 
  | neb://ipp047.0/gpc1/20111126/o5891g0257o/o5891g0257o.ota67.fits |
  

Tuesday : 2013.09.03

Bill is czar today

• 07:15 registration is stuck. Can't see why everybody has burntool_state = -14. Restarting pantasks.
• 07:25 Ah there it is. burntool was stuck because one chip had burntool_state = -1 and data_state = check_burntool. too early in the morning.
• forgot to mention, ipp028 claimed to be down in ganglia but was up. It looks like it restarted around 6am today. It is now showing up in ganglia
• 09:30 We got reports of slow access to the data store last night. This may have been related to the ipp028 outage, but I haven't investigated. For good measure though I have restarted apace on ippc17.
• 09:35 nightly science chips are done. Setting chip.revert.off to avoid because there are 90 or so LAP chips that are faulting due to ipp047 problems.
  • MEH -- there is a tweak_chiprevert file to turn off chip.revert except for a couple times during nightly processing for when stdsci is restarted
• 11:49 restarted update pantasks earlier but forgot to setup and run. fixed that
• 12:10 restarted cleanup pantasks.
  • Testing cleanup of old distRuns with outroot like '/%' (non-nebulous) diff.cleanup off for now while I make sure cleanup of old directories doesn't raise havoc. Looks good so far.
• 12:01 restarted stdscience but forgot to mention it. diff cleanup started up again
• 15:45 MEH: as discussed last meeting, all MD04 being updated this week (label MD04.pv2.20130903). will be watching, so no need to revert etc
• 18:18 removed label goto_cleaned.rerundiff label from cleanup. Since the diffRuns with that label are older and being rerun,there is less space to recover than more recent (higher diff_id) runs which have images to clean up.
• 19:00 MEH: ippdb02 and ippdb04 are still increasing sec behind master, pausing cleanup and stdscience before nightly to see if decreases a bit
• 19:30 MEH: ippdb04 slowly catching up ~2ks, ippdb02 still 110900s behind (suspect doing backup still) -- leaving cleanup off for a while longer, LAP label out, nightly starting
  • with nightly running, ippdb04 barely steady ~22k behind, turning cleanup and LAP back on
  • iperf running on ippdb04 @400% CPU.. kill -STOP and see if helps (running since 7/30, since last boot)
• 19:31 turned chip.revert.on
  • MEH -- and set the tweak_chiprevert so will only activate ~midnight and 4am

Wednesday : 2010.09.04

• 04:30 reverted faulted about 6 nightlyscience faulted chips by hand
• 04:34 chip stage is done for 221 new exposures
• 07:50 MEH: ippdb04 has at least caught up, starting MD04.pv2 again

Thursday : 2010.09.05

Friday : 2010.09.06

Saturday : 2010.09.07

Sunday : 2010.09.08