== PS1 IPP Czar Logs for the week 2015.05.04 - 2015.05.10 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.05.04 ===
 

 * ippc18 is off - this causes cron jobs to not run as well as ganglia. (so this is a wrning)
 * haf : 21:00 registration sucks:
{{{
    21	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY37 -set_state pending_burntool -dbname gpc1
    22	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY42 -set_state pending_burntool -dbname gpc1
    23	20:53	regtool -updateprocessedimfile -exp_id 909463 -class_id XY62 -set_state pending_burntool -dbname gpc
    32	21:50	regtool -updateprocessedimfile -exp_id 909566 -class_id XY37 -set_state pending_burntool -dbname gpc1
}}} 
=== Tuesday : 2015.05.05 ===
 * 03:50 Bill: Registration is falling behind. There are 3 register_imfile.pl processes that have been running for 10 hours. I am beginning to investigate.
  * those processes were not an issue They were all from one dark exposure that had already finished registration. The scripts must have got stuck while exiting.
  * However exp_id 909635  XY31 was stuck in check_burntool data_state. Set it back to pending_burntool and registration is now catching up.
 * 05:40 Bill: registration is stuck again 3 apply_burntool single processes have been running for > 6000 seconds XY42, XY75, and XY17. The hosts running the processes ipp004, ipp010, and ipp054 are the same ones that were stuck previously. There is probably an nfs problem.
  * killed the burntool program processes that were inexplicably stuck and the system fixed the burntool_state on it's own.
  * set hosts ipp004, ipp010, and ipp054 to off in registration pantasks. They seem to keep hanging.
 * 12;29 Bill: set pstamp pantasks to stop in preparation for a restart. Lots of faults in the status that are distracting.
  * 12:36 restarted pstamp pantasks
 * 12:40 CZW: restarting pv3fflt/rt.
 * 20:12 HAF: registration stuck
   * {{{ regtool -updateprocessedimfile -exp_id 909868 -class_id XY02 -set_state pending_burntool -dbname gpc1}}}
   *  {{{regtool -updateprocessedimfile -exp_id 909877 -class_id XY06 -set_state pending_burntool -dbname gpc1}}}
   *
 {{{ 
regtool -updateprocessedimfile -exp_id 909885 -class_id XY06 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 909887 -class_id XY06 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 909890 -class_id XY06 -set_state pending_burntool -dbname gpc1

}}}
 * 21:35 HAF : ipp010 / ipp054 / ipp029 had stck mounts, gene fixed them (mostly) - ipp010 , ipp029 are fixed, ipp054, ippc19 are unhappy still, but better than we were..
 * 00:45 MEH: registration >200 behind, doesn't look like any gpc1 data has processed in a while (though has been doing gpc2). seems to be stuck on 
{{{
o7148g0073o  XY06 -14 pending_burntool neb://ipp091.0/gpc1/20150506/o7148g0073o/o7148g0073o.ota06.fits  
}}}
  * looks like this is full but got set to pending -- doing 
{{{
regtool -updateprocessedimfile -exp_id 909890 -class_id XY06 -set_state full -dbname gpc1
}}}
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===