== PS1 IPP Czar Logs for the week 2011-06-20 - 2011-06-26 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011-06-20 ===
 * 09:05 (Serge) Reverted 31 unpublished 3Pi exposures from yesterday

=== Tuesday : 2011-06-21 ===

 * 14:00 CZW: Rebuilt new ipp-20110621 tag to include stack rejection improvements.
 * 15:00 CZW: Restarted processing with LAP Stripe 82 data.
 * 22:49 CZW: Discovered that there seems to be a bug in the asinh scaling of expwt images in the LAP Stripe 82 test, causing lots of faults. It's probably just a bad sigma value that isn't being caught correctly, so I'll let the stacks that are failing stay faulted until I can fix it tomorrow.
 * 22:57 CZW: Queued first 20 month 0, g.00000 band LAP data for lapSequence 6 (this is the new LAP reprocessing).  This should keep the cluster busy until tomorrow when I can fix the asinh bug.
=== Wednesday : 2011-06-22 ===
 * 11:20 bill: re-ran chip XY20 from chipRun 245052 which had a corrupted fits image (on ipp013)
 * 16:57 CZW: stopping processing to change to new-new tag (ipp-20110622) which incorporates Gene's changes.
 * 17:35 CZW: bringing system back online on new tag.
=== Thursday : 2011-06-23 ===
 * 09:xy serge reverted publish faults
 * 09:xy heather reverted cam/stack/warp/diff faults
 * 13:00 bill set several publishRuns to drop. All of the inputs had non-zero quality. publish_run.pl doesn't deal with that case properly.
 * 13:20 bill set chip 246194 XY17 to quality 42. It was faulting repeatedly.
 * 13:25 bill reverted 3 lap stacks which apparently got all the way to the end before faulting writing the output files.
 * 15:02 serge reverted publish faults
 * 16:00 heather removed LAP.ThreePi.20110621 to speed up nightlyscience processing. will add it back in once nightly processing  is done.
 * 17:00 heather added back in the LAP label - there is one skycell left of one diff, and it's been processing for 1000 seconds. If it doesn't finish soon I'll do something about that...
 * 17:14 heather had to 'fix' a few of the diff skycells - they had been trying to process for long periods of time (since morning). Heather killed the ppSub process and reverted them. That seems to work.
=== Friday : 2011-06-24 ===
 * 06:20 serge reverted faulty publications: pubtool -revert -dbname gpc1 -fault 2
 * 06:21 serge set LAP label to inactive: labeltool -updatelabel -set_inactive -label LAP.20110621 -dbname gpc1
 * 06:46 CZW: add ESS/MSS targets to nightly_science.config. Queue missing ESS data from last night with: {{{chiptool -simple -dbname gpc1 -definebyquery -set_end_stage warp -set_label SweetSpot.nightlyscience -set_workdir neb://@HOST@.0/gpc1/ESS.nt/20110624 -set_dist_group SweetSpot -set_tess_id RINGS.V0 -set_data_group ESS.20110624 -obs_mode ESS -dateobs_begin 2011-06-24T00:00:00 -dateobs_end 2011-06-24T23:59:59}}}
 * 08:24 heather removed LAP.ThreePi.20110621 for the nightly science to finish
 * 08:58 bill changed the SweetSpot priority to 410 (larger than ThreePi)
 * 10:05 serge set chip.off to speed up ESS processing
 * 11:20 chip.on since ESS processing is stuck
 * 12:00 chip.off again since diffs have been scheduled
 * 13:12 chip.on


=== Saturday : 2011-06-25 ===

=== Sunday : 2011-06-26 ===