PS1 IPP Czar Logs for the week 2011-06-20 - 2011-06-26

(Up to PS1 IPP Czar Logs)

Monday : 2011-06-20

• 09:05 (Serge) Reverted 31 unpublished 3Pi exposures from yesterday

Tuesday : 2011-06-21

• 14:00 CZW: Rebuilt new ipp-20110621 tag to include stack rejection improvements.
• 15:00 CZW: Restarted processing with LAP Stripe 82 data.
• 22:49 CZW: Discovered that there seems to be a bug in the asinh scaling of expwt images in the LAP Stripe 82 test, causing lots of faults. It's probably just a bad sigma value that isn't being caught correctly, so I'll let the stacks that are failing stay faulted until I can fix it tomorrow.
• 22:57 CZW: Queued first 20 month 0, g.00000 band LAP data for lapSequence 6 (this is the new LAP reprocessing). This should keep the cluster busy until tomorrow when I can fix the asinh bug.

Wednesday : 2011-06-22

• 11:20 bill: re-ran chip XY20 from chipRun 245052 which had a corrupted fits image (on ipp013)
• 16:57 CZW: stopping processing to change to new-new tag (ipp-20110622) which incorporates Gene's changes.
• 17:35 CZW: bringing system back online on new tag.

Thursday : 2011-06-23

• 09:xy serge reverted publish faults
• 09:xy heather reverted cam/stack/warp/diff faults
• 13:00 bill set several publishRuns to drop. All of the inputs had non-zero quality. publish_run.pl doesn't deal with that case properly.
• 13:20 bill set chip 246194 XY17 to quality 42. It was faulting repeatedly.
• 13:25 bill reverted 3 lap stacks which apparently got all the way to the end before faulting writing the output files.
• 15:02 serge reverted publish faults
• 16:00 heather removed LAP.ThreePi.20110621 to speed up nightlyscience processing. will add it back in once nightly processing is done.
• 17:00 heather added back in the LAP label - there is one skycell left of one diff, and it's been processing for 1000 seconds. If it doesn't finish soon I'll do something about that...
• 17:14 heather had to 'fix' a few of the diff skycells - they had been trying to process for long periods of time (since morning). Heather killed the ppSub process and reverted them. That seems to work.

Friday : 2011-06-24

• 06:20 serge reverted faulty publications: pubtool -revert -dbname gpc1 -fault 2
• 06:21 serge set LAP label to inactive: labeltool -updatelabel -set_inactive -label LAP.20110621 -dbname gpc1
• 06:46 CZW: add ESS/MSS targets to nightly_science.config. Queue missing ESS data from last night with: chiptool -simple -dbname gpc1 -definebyquery -set_end_stage warp -set_label SweetSpot.nightlyscience -set_workdir neb://@HOST@.0/gpc1/ESS.nt/20110624 -set_dist_group SweetSpot -set_tess_id RINGS.V0 -set_data_group ESS.20110624 -obs_mode ESS -dateobs_begin 2011-06-24T00:00:00 -dateobs_end 2011-06-24T23:59:59
• 08:24 heather removed LAP.ThreePi.20110621 for the nightly science to finish
• 08:58 bill changed the SweetSpot priority to 410 (larger than ThreePi)
• 10:05 serge set chip.off to speed up ESS processing
• 11:20 chip.on since ESS processing is stuck
• 12:00 chip.off again since diffs have been scheduled
• 13:12 chip.on

Saturday : 2011-06-25

Sunday : 2011-06-26