== PS1 IPP Czar Logs for the week 2011.09.19 - 2011.19.25 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.09.19 ===
  
 * 08:00: Roy:  everything is down from the summit.
 * 08:30: Roy: {{{ipp033}}} is back, so ran:

{{{
neb-host ipp033 up
}}}
 * 09:15: Bill: Postage stamp server is caught up. Set all data with labels ps_ud% to be cleaned.
 * 09:20: Bill: cleared all failed_revert faults.
 * 10:10: Bill: set lap priority to 410 to allow it to finish it's destreak.
 * 11:00: Bill: set lap priority back to 200
 * 12:10 Chris/Mark: stdscience going slow. pcontrol large CPU use, stopping panstasks for restart.
 * 12:29: Roy: shutdown {{{stdscience}}}, and restarted it
 * 15:28: Bill fixed bug in ppMops that was causing some faults.
 * 16:06: Bill fixed bug in his STS queueing script. Ran it to queue diffs for Sept 5 which were blocked by the bug
 * 18:00: Mark removed labels from stdscience not being used tonight, MD03,04,05,06,07,08, PI, STD.nightlyscience, to test processing hiccups.
 * 20:10 few frames of MD08 were observed, adding it back in.
=== Tuesday : 2011.09.20 ===

 * 08:35: Roy: registration stuck. {{{regpeek}}} revealed:

{{{
o5824g0388o  XY11 0 check_burntool neb://ipp008.0/gpc1/20110920/o5824g0388o/o5824g0388o.ota11.fits
}}}

Got exp_id using

{{{
SELECT DISTINCT exp_id FROM rawImfile WHERE exp_name = 'o5824g0388o';
+--------+
| exp_id |
+--------+
| 395112 | 
+--------+
}}}

Then ran

{{{
regtool -updateprocessedimfile -exp_id 395112 -class_id XY11 -set_state pending_burntool -dbname gpc1
}}}

 * 08:40: Roy: Same as above, this time for exposure {{{o5824g0398o}}}
 * 08:42: Roy: And again, exposure {{{o5824g0399o}}}
 * 8:48 heather redid one of the failed burntool commands in the pantasks log.
 * 9:00 CZW: restarted distribution server as it had the 100% CPU pantasks, and was processing slowly.
 * 09:45 Bill: update server was down. Restarted it
 * 10:19 Started magic_cleanup server in order to recover disk space on ipp053 ~bills/magic_cleanup.
 * 14:47 CZW: Stopped stdscience to restart it.
=== Wednesday : 2011-09-21 ===
 * 1:20 CZW: Stopped distribution to restart it.
 * 18:20 CZW: Preparation for stare night:
   * Stopped stdscience, distribution, cleanup, stack pantasks to prevent any processing 
   * Disabled stare node processing using script (~ipp/stare_nodes.sh off)
   * Disabled future stare node processing by editing ~ipp/ippconfig/pantasks_hosts.input
   * Disabled nebulous allocation to all but the most full hosts using script: ~ipp/stare/pre_stare_night
=== Thursday : 2011-09-22 ===
 * 10:25 Stopped pantasks. Set nebulous state back to normal. Began restarting the pantasks.
 * 10:36 Fixed warp with missing psf file runwarpskycell.pl --warp_id 260390 --skycell_id skycell.0783.030. Reverted the stack that wanted it.
 * 10:50 Oops summit copy is only about 1/2 done with last night's data. Set nebulous back to pre_stare_night. Also rerunning the warp didn't take. Strange process, cleanup, update yields bad quality problem.

=== Friday : 2011-09-23 ===

=== Saturday : 2011-09-24 ===

=== Sunday : 2011-09-25 ===