== PS1 IPP Czar Logs for the week 2012.06.11 - 2012.06.18 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.06.11 ===
  * 21:20 leaving postage stamp server running along with nightly science. seems to do fine with active MOPS jobs past couple nights while being monitored, but will turn update off later. some QUB stamps seem to have stalled even with update on and no other stamps running, will need looking into. 
=== Tuesday : 2012.06.12 ===
 Mark is czar
 * 07:20 nightly science data downloaded and processed. postage stamp running still, MOPS jobs almost finished but more QUB jobs seems to have become stuck and am looking into.
 * 09:20 tracing out the PSS stalling, one looks like wanting/waiting to update a diffim that is in the goto_cleaned state (never actually cleaned), MD06 diff_id=242933
  * turning on cleanup pantasks and watching processing/ippdb03 closely to avoid crashing, try and "clean-up" -- looks like will take a couple hours. want to turn back off before nightly science? 
  * 12:30 QUB from past few days moving through now as well as some web ones too.
 * 10:00 Serge turning replication/balance back on since doesn't use gpc1
 * 10:30 removing ippc12 from pantasks and from loaded compute list since being swapped in for ippdb01.
 * 10:35 did ippc12 going down stall a distribution entry for MD07 SSdiff?
  * do the easy restart of distribution and see if picks up again -- yes and cleared.
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===