== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : YYYY.MM.DD ===
  
 * 11:25 CZW: started pantasks server on j001b09 to do raw data shuffle to backup nodes.
=== Tuesday : 2014.11.11 ===

  * 5:00 HAF registration is stuck on o6972g0517o -- usual bag o tricks didn't work. I ended up setting a quality of 42 on that stupid burntool. {{{update rawImfile set quality = 42, fault = 0, state = 'full' where exp_name = 'o6972g0517o' and class_id = 'XY14';}}}, but not after spending an hour trying to figure out what was going on...
  * 6:20 HAF registration is still stick on teh same exp, after moving on.. I redid the update since it seems to have been reverted... registration is moving again but I don't trust it
  * 6:24 Chris investigated, regtool -updateprocessedimfile -burntool_state -14 -exp_id 816417 -class_id XY14 -set_ignored -set_state full is the proper way to tell the database to ignore an xy
 * 12:00 MEH: mops wondering where data is, stdsci looks to have been down since ~10am? did someone shutdown? 
  * restarting .. -- lanl stdlocal off until nightly finished (wasnt doing anything anyways).. putting ippsXX nodes into stdsci to finish up asap..
 * 12:40 MEH: ipp035 unresponsive.. neb-host down until back up, out of processing -- 
 * 12:50 MEH: ipp035 up, putting back into repair -- ipp029 now unresponsive.. seriously.. 
  * ipp029 from neb-host up to down -- out of processing
  * ipp029 up (kernel panic), putting neb-host repair (normally up)

 * 13:40 MEH: when nightly finished, '''ippsXX off in stdsci and stdlocal for MD/CNP processing''', stdlocal on again and update labels back into stdsci
 * 15:30 EAM : stdlocal is going slowly, so i'm stopping stacks in prep for restarting
  * good was inbetween runs, had set the ippsXX off in stdlocal.. setting back to off again
 * 17:30 HAF : restarted registration, it had crashed (? i assumed?)

=== Wednesday : 2014.11.12 ===

 * 06:40 EAM : I am stopping stdlocal for now in anticipation of launching relphot on the compute nodes.  Also note that nightly science is proceeding slowly - is this the db again?
   * 10:32 HAF : in answer to Gene's question - I noticed that we were slow to download images last night - we did catch up, and probably without errors (?). I had noticed timeouts on Tuesday, so I wonder if that's related (ie, db is slow?)
 * 07:20 EAM : lots of warps still to be done -- I've added compute nodes c2 to stdscience to up the throughput.

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===