PS1 IPP Czar Logs for the week 2011-06-20 - 2011-06-26

(Up to PS1 IPP Czar Logs)

Monday : 2011-06-20

  • 09:05 (Serge) Reverted 31 unpublished 3Pi exposures from yesterday

Tuesday : 2011-06-21

  • 14:00 CZW: Rebuilt new ipp-20110621 tag to include stack rejection improvements.
  • 15:00 CZW: Restarted processing with LAP Stripe 82 data.
  • 22:49 CZW: Discovered that there seems to be a bug in the asinh scaling of expwt images in the LAP Stripe 82 test, causing lots of faults. It's probably just a bad sigma value that isn't being caught correctly, so I'll let the stacks that are failing stay faulted until I can fix it tomorrow.
  • 22:57 CZW: Queued first 20 month 0, g.00000 band LAP data for lapSequence 6 (this is the new LAP reprocessing). This should keep the cluster busy until tomorrow when I can fix the asinh bug.

Wednesday : 2011-06-22

  • 11:20 bill: re-ran chip XY20 from chipRun 245052 which had a corrupted fits image (on ipp013)
  • 16:57 CZW: stopping processing to change to new-new tag (ipp-20110622) which incorporates Gene's changes.
  • 17:35 CZW: bringing system back online on new tag.

Thursday : 2011-06-23

  • 09:xy serge reverted publish faults
  • 09:xy heather reverted cam/stack/warp/diff faults
  • 13:00 bill set several publishRuns to drop. All of the inputs had non-zero quality. publish_run.pl doesn't deal with that case properly.
  • 13:20 bill set chip 246194 XY17 to quality 42. It was faulting repeatedly.
  • 13:25 bill reverted 3 lap stacks which apparently got all the way to the end before faulting writing the output files.
  • 15:02 serge reverted publish faults
  • 16:00 heather removed LAP.ThreePi?.20110621 to speed up nightlyscience processing. will add it back in once nightly processing is done.
  • 17:00 heather added back in the LAP label - there is one skycell left of one diff, and it's been processing for 1000 seconds. If it doesn't finish soon I'll do something about that...
  • 17:14 heather had to 'fix' a few of the diff skycells - they had been trying to process for long periods of time (since morning). Heather killed the ppSub process and reverted them. That seems to work.

Friday : 2011-06-24

  • 06:20 serge reverted faulty publications: pubtool -revert -dbname gpc1 -fault 2
  • 06:21 serge set LAP label to inactive: labeltool -updatelabel -set_inactive -label LAP.20110621 -dbname gpc1
  • 06:46 CZW: add ESS/MSS targets to nightly_science.config. Queue missing ESS data from last night with: chiptool -simple -dbname gpc1 -definebyquery -set_end_stage warp -set_label SweetSpot.nightlyscience -set_workdir neb://@HOST@.0/gpc1/ESS.nt/20110624 -set_dist_group SweetSpot -set_tess_id RINGS.V0 -set_data_group ESS.20110624 -obs_mode ESS -dateobs_begin 2011-06-24T00:00:00 -dateobs_end 2011-06-24T23:59:59
  • 08:24 heather removed LAP.ThreePi?.20110621 for the nightly science to finish
  • 08:58 bill changed the SweetSpot? priority to 410 (larger than ThreePi?)
  • 10:05 serge set chip.off to speed up ESS processing
  • 11:20 chip.on since ESS processing is stuck
  • 12:00 chip.off again since diffs have been scheduled
  • 13:12 chip.on
  • 18:53 bill reruning --diff_id 140442 --skycell_id skycell.101 which left a corrupt file lying around. This should allow the last md07 diff run to finish
  • 18:56 add.label LAP.ThreePi?.20110621 to stdscience
  • 19:04 bill turned stack.revert.on in the stack pantasks. All of the faults that we have had recently have been file access faults.
  • 19:13 bill queued diffRuns for STS.20110624.

Saturday : 2011-06-25

  • 07:10 serge:
    del.label LAP.ThreePi.20110621
    
  • 08:00 serge:
    add.label LAP.ThreePi.20110621
    
  • 18:00 Bill: an sts chip destreak run was failing repeatedly. At the censor sources stage it streaksremove was encountering sources with x and y == NAN. That is bad form and a psphot bug. I modifed streaksremove to censor sources like this.
  • 20:00 Bill: I fixed a corrupt diff file that was causing a 3pi magic failure. And I turned on publish revert since the failures lately have been transient nfs faults.

Sunday : 2011-06-26