== PS1 IPP Czar Logs for the week 2010.09.02 - 2010.09.08 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.09.02 ===
 * 00:10 MEH: looks like registration is stuck on something, 36 behind
 * 11:30 MEH: Serge reported a warp continues faulting, looks like no good sources (1 psf source) -- edge skycell so set quality 42 for now.. exposure publishing for MOPS now
{{{
warptool -dbname gpc1 -updateskyfile -warp_id 836928 -skycell_id skycell.049 -fault 0 -set_quality 42
}}}
 * 11:40 MEH: LAP stalled, good time for regular restart stdsci and clearing all LAP chip faults so processing can resume..
 * 12:10 MEH: MD01 warp qual held up stacks and SSdiff as well -- stacks finished and tweak_ssdiff to run 
 * 19:10 MEH: LAP near run out of chips again, ~50 to fix. more single chips only on ipp047 -- 5 more to the list for 17
{{{
| neb://ipp047.0/gpc1/20111120/o5885g0116o/o5885g0116o.ota67.fits | 
| neb://ipp047.0/gpc1/20111125/o5890g0407o/o5890g0407o.ota67.fits | 
| neb://ipp047.0/gpc1/20111126/o5891g0203o/o5891g0203o.ota67.fits | 
| neb://ipp047.0/gpc1/20111126/o5891g0117o/o5891g0117o.ota67.fits | 
| neb://ipp047.0/gpc1/20111126/o5891g0127o/o5891g0127o.ota67.fits | 
| neb://ipp047.0/gpc1/20111126/o5891g0222o/o5891g0222o.ota67.fits | 
| neb://ipp047.0/gpc1/20111125/o5890g0419o/o5890g0419o.ota67.fits | 
| neb://ipp047.0/gpc1/20111120/o5885g0127o/o5885g0127o.ota67.fits | 
| neb://ipp047.0/gpc1/20111120/o5885g0190o/o5885g0190o.ota67.fits | 
| neb://ipp047.0/gpc1/20111125/o5890g0412o/o5890g0412o.ota67.fits | 
| neb://ipp047.0/gpc1/20111120/o5885g0175o/o5885g0175o.ota67.fits | 
| neb://ipp047.0/gpc1/20111126/o5891g0226o/o5891g0226o.ota67.fits | 
| neb://ipp047.0/gpc1/20111126/o5891g0228o/o5891g0228o.ota67.fits | 
| neb://ipp047.0/gpc1/20111125/o5890g0508o/o5890g0508o.ota67.fits | 
| neb://ipp047.0/gpc1/20111125/o5890g0496o/o5890g0496o.ota67.fits | 
| neb://ipp047.0/gpc1/20111126/o5891g0260o/o5891g0260o.ota67.fits | 
| neb://ipp047.0/gpc1/20111126/o5891g0257o/o5891g0257o.ota67.fits |
}}}
=== Tuesday : 2013.09.03 ===
Bill is czar today
 * 07:15 registration is stuck. Can't see why everybody has burntool_state = -14. Restarting pantasks.
 * 07:25 Ah there it is. burntool was stuck because one chip had burntool_state = -1 and data_state = check_burntool. too early in the morning.
 * forgot to mention, ipp028 claimed to be down in ganglia but was up. It looks like it restarted around 6am today. It is now showing up in ganglia
 * 09:30 We got reports of slow access to the data store last night. This may have been related to the ipp028 outage, but I haven't investigated. For good measure though I have restarted apace on ippc17.
 * 09:35 nightly science chips are done. Setting chip.revert.off to avoid because there are 90 or so LAP chips that are faulting due to ipp047 problems.
  * MEH -- there is a tweak_chiprevert file to turn off chip.revert except for a couple times during nightly processing for when stdsci is restarted
 * 11:49 restarted update pantasks earlier but forgot to setup and run. fixed that
 * 12:10 restarted cleanup pantasks. 
  * Testing cleanup of old distRuns with outroot like '/%' (non-nebulous) diff.cleanup off for now while I make sure cleanup of old directories doesn't raise havoc. Looks good so far.
 * 12:01 restarted stdscience but forgot to mention it. diff cleanup started up again
 * 15:45 MEH: as discussed last meeting, all MD04 being updated this week (label MD04.pv2.20130903). will be watching, so no need to revert etc
 * 18:18 removed label goto_cleaned.rerundiff label from cleanup. Since the diffRuns with that label are older and being rerun,there is less space to recover than more recent (higher diff_id) runs which have images to clean up.
 * 19:00 MEH: ippdb02 and ippdb04 are still increasing sec behind master, pausing cleanup and stdscience before nightly to see if decreases a bit
 * 19:30 MEH: ippdb04 slowly catching up ~2ks, ippdb02 still 110900s behind (suspect doing backup still) -- leaving cleanup off for a while longer, LAP label out, nightly starting
=== Wednesday : 2010.09.04 ===

=== Thursday : 2010.09.05 ===

=== Friday : 2010.09.06 ===

=== Saturday : 2010.09.07 ===

=== Sunday : 2010.09.08 ===