Summary ---IN PROGRESS PLEASE DO NOT EDIT---

Discovered for many inputs into ppStack (i.e., deep and even refstacks), that the PSF in the output convolved stack grows with the number of inputs beyond the initial target PSF. Had been missed due to convolved stacks really only being used for magic, but was noticed with some M31 analysis and deepstacks of MD04. Memory use has also been an issue with the deepstacks and simultaneously being investigated.

Example: some MD04 skycell.055, ~100 input warps

  • FWHM input average from log = 4.48+-0.81 (pixels)
  • largest input has FWHM_MAJ of ~7.2
  • target FWHM from log 6.88, Kron~6.29 from convolution part of log
  • convolved stack FWHM_MAJ=14.36
  • for comparison, unconvolved stack FWHM_MAJ=4.32
left to right: input warp FWHM_MAJ~6.8, unconvolved stack, convolved stack

More specific examples:

  • MD04.refstack.20111012 skycell.055-i ~92 inputs
          Input FWHMs : 4.010976 +/- 0.334657
              fit ext:   1.439042 sec for 225 of 225 sources
              fit psf:   0.395260 sec for 172 of 225 sources
            apresid: 0.022497 +/- 0.005780 (systematic error) from statistics of 225 psf stars (172 used)
            chisq vs flux fit term 0: 0.000000 +/- 0.000000
            chisq vs flux fit term 1: 0.000000 +/- 0.000000
          try model PS_MODEL_PS1_V1, ap-fit: 0.022497 +/- 0.005780 : sky bias: 0.000000
          Target seeing FWHM: 4.706741
    --> output convolved PSF, end of log
            fwhm (psf): 10.825166,10.697605 (moments): 0.000000,0.000000
    
  • MD04.deeptest.20111012 skycell.055-i ~300 inputs
          Input FWHMs : 4.835018 +/- 0.719810
              fit ext:   1.385216 sec for 225 of 225 sources
              fit psf:   0.441030 sec for 171 of 225 sources
            apresid: 0.055969 +/- 0.001243 (systematic error) from statistics of 225 psf stars (171 used)
            chisq vs flux fit term 0: 0.000000 +/- 0.000000
            chisq vs flux fit term 1: 0.000000 +/- 0.000000
          try model PS_MODEL_PS1_V1, ap-fit: 0.055969 +/- 0.001243 : sky bias: 0.000000
          Target seeing FWHM: 6.630503
    --> output convolved PSF, end of log
            fwhm (psf): 14.636683,14.511685 (moments): 0.000000,0.000000
    

Status

  • ...
  • 6/19 restarting to look at target PSF, merged trunk (r34040) to pptack_test branch again, so tests (target PSF) now will be using that
  • 6/20 added simple PSF model option to use mean FWHM + X for Target PSF for testing
  • 6/21 finally brought page up to date with notes.. reset testing directories for target PSF tests and re-ran initial simtest set.
  • 6/28 revised sim run based on results/plots from 6/21.

Other Issues/Problems

Will add to as things are found/discussed:

  • (fixed) hard limit on number of inputs
    • was shell descriptors limit of 1024 easily being reached with image,mask,weight,psf,cmf files for each input. Raised to 65535 now by default in shells.
  • (in progress) selection of target PSF FWHM (for convolved stack)
    • worse input PSF not necessarily best choice as could end up being rejected and only a few inputs may be that large (not representative)
  • (on hold) heavy RAM use
    • 10-20GB typical for 100+ inputs -- tests run unoptimized, will need to look at effect
  • (on hold) long run-time for 20+ inputs
    • 400 inputs order 12-14hr (use method similar to psphotStack?) -- tests run unoptimized, will need to look at effect
  • (on hold) stacks (nightly and deeper) seeing more of the fault 5 stack events
    • for MD nightly where inputs have strangely small <2 pixel input PSF.
    • Bill has code for quality setting but not turned on, more of an issue possibly with the PSF determination for edge skycells with possible poor extrapolation from center of the field.
    • Not really a ppStack issue but needs to deal with and not fault.
  • convolved stacks have different background than unconvolved stacks?

Code/Branch Modification

Branch for testing -- http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/browser/branches/meh_branches/ppstack_test

  • originally r32161 from last summer -- needed to update --
    • merge on 20120305 of r33407 trunk -- most of conv PSF fixes here
    • merge on 20120619 of r34040 trunk -- working on Target PSF here
  • many outputs turned on (runs slower, makes many many files)
    • ppStackMatch.c -- need add a TESTING2 for reading in already processed conv images..
    • ppSubMatchPSFs.c has TESTING but not ifdef at start, is a .DEBUG. used to trigger?

Now all modification tests using updated branch from trunk/ipp-20120531 (r34040, 6/19/2012) -- Target PSF for conv stacks (some modifications still not added below yet, mostly TESTING?)

  • ppStackTarget.c -- Gene found bug with meanVariance after merge, so added
  • simple model PSF as target PSF set by input FWHM for new tag ipp-20120626
    • ppStackOptions.c/h -- added clippedMean/Stdev floats
    • ppStack.h -- ppStackPSF for all option not just option Mask
    • ppStackPrepare.c -- setting the option->clippedMean/stdev
    • ppStackPSF.c -- use the clippedMean and stdev if options for it
    • recipes/ppStack.config -- added keyword for +Nsig option

All modification tests now using updated branch from ipp-20120216 tag (r33407, 3/04/2012) -- testing modifications, some added to trunk already

  • pmSubtractionMatch.c:
    • has isfinite(sysError) rather than skyError compared to 0 - fixed +trunk
    • sbrk in memCheck issue on OSX - not sure how to fix, added note +trunk
  • pmSubtraction.c:
    • TESTING for output conv1/2_* has problem with index now -- what supposed to be? fix removed index and add comment +trunk
    • comment ConvolveStampThread? footprint as stamp index is wrong -- fix +trunk
  • ppStackMatch.c
    • TESTING w/ and w/o using previous conv images or output samples.. need to split the two -- more explicit TESTING_REUSE_CONV case added - fix +trunk
    • Bill found MASK.BAD/VAL issue in psphotStack also will be here in BG model, no clear problem noticed here -- fix local (should merge from trunk to pick up other fixes)

  • ppStackSetup.c -
    • also conflict issue with names ZP is active, MATCH.ZERO.POINTS inactive for recipes/ppStack.config -- leaving, not sure how fix
  • ppStackSources.c -
    • logic issue with ZP and photometry option, if ZP false then segfault - false not work, need to be OR? -- need to trace out
    • also TESTING mix again - outputs and fake added mag offsets -- wait
    • TESTING section for start_#.fits not possible now with order change in ppStackPrepare.c -- local and turned off now
    • if PHOTOMETRY and many rejected from poor detections or in past more commonly magic masking on edges, then Abort case triggered since minMatches WRT all inputs and not numGoodImages -- fix local
  • ppStackConvolve.c calls with normal index but ends up odd index in kernel name - bug?
    • .kernel index odd, why not seq in pmFPAfileActivateSingle - psMod/camera/pmFPAfileIO.c -- SelectSingle? in pmFPAfile.c -- could use date stamp or is threading the reason?
  • pmSubtractionStamps.c
    • the stamp.dat output condition case is trouble, if no sources in a stamp will segfault -- set to 0 again.. -- wait
  • recipes/ppSub.config
    • set SYS.ERR in the STACK metedata -- ALL stacks will use this then -- fix +trunk +ipp-20120216 tag (3/16 UT)
  • ppStackPrepare.c
    • order logic for reject masking images changed, since target PSF uses all "good" images, all rejections should be done before but zero point/transpanency calcs for each image were done after. Moved above target PSF section, this makes the TESTING section in ppStackSources.c for start_#.fits images not possible since uses the target PSF to make. -- fix local
  • pmReadoutFake.c
    • typecast S32/F32 conflict resulting in 0 value wrongly setting stamp sizes when calling pmReadoutFakeFromSources -- fix local +trunk+20120404 tag included
  • psLib/somewhere
    • needs to be fixed if -Db option used that it fails like others in that a config option not provided and not silently go back and use one from default config files
  • ippTools/src/stacktool.c,
    • more options for definebyquery likely necessary to dump problem input images (ellipticity immediately comes to mind..) - added locally only, added to trunk/tag too

All testing currently being dumped into /data/ipp059.0/mhuber/testppstack_deep201111/ which has reached >3TB and will need to probably clean up some soon (>2TB in 400 stk grizy test)

  • moved to underused disk /data/ipp060.0/mhuber/testppstack_deep201111 (june 2012)

Test Sets

More easily setup than for testing ppSub, need a large number of various inputs and some number of similar inputs.

MD04 (GR0)

Sample of 20 and 100 drawn from the the MD04.V3 GR0 deepstack set for memory testing as well.

  • FWHM in ppStack logs, cut and put into histogram below. Need to compare to psphot FWHM_MAJ/IQ_FW1?

Batch scripts for running in /data/ipp059.0/mhuber/testppstack_deep201111/MDdeeptest

Simulated (simtest)

A small variety set of four with many groups to make 20 and 100 inputs sets

Simtest setup (mostly default):

  • default pixelscale ~0.26"/pixel, exptime 20s, filter r, 1k x 1k
  • PSF.MODEL PS_MODEL_GAUSS
filename sim FWHM FWHM_MAJ (pixel) IQ_FW1 (pixel)
image.0 1.0 3.82 2.98
image.1 1.1 4.19 3.28
image.2 1.2 4.58 3.62
image.3 1.5 5.69 4.49
_and repeat_

Batch scripts for running in various /data/ipp059.0/mhuber/testppstack_deep201111/ppstack_simdeep* (needs cleanup)


Initial Tests

  • do clean simulated images suffer the same problem?
    • can use for more detailed testing since faster to run than MD04 skycell and not particular to GPC1 data
  • is it related to the range of inputs and number
    • is 20 inputs enough to reveal problem, if 100 then will be tedious
  • what are the output convolved images being stacked, what is the target the images are being convolved to
  • need to understand what the code does, so first test is to look at each output, add others and step through the code...
  • tests indicate convolution of each input seem to be larger than expected -- appears to be also with the range of input FWHM

MD04: 20 input case:

  • initially run for a memory test, but added extra outputs and some missing
  • input FWHM 4.49+/-0.48 (mean)
  • target seeing FWHM: 5.06, kron ~4.0
filename FWHM_MAJ (pixel) IQ_FW1 (pixel) .log Input FWHM (pixel) .log kron (pixel)
real_000 5.38 4.20 5.40 4.25
real_002 5.08 4.02 5.03 4.24/33
real_009 4.27 3.17 4.19 3.42/33
fake_000 5.10 4.09
fake_002 5.13 4.11
fake_009 5.10 4.09
conv.0 x x
conv.2 8.41 6.37
conv.9 7.91 6.02
comb.initial x x
stack 8.39 6.06 (also initially missing from header)
stack.unconv 4.02 3.55

MD04: 100 input case:

  • was run for memory monitoring, will need to be rerun for real/fake/conv outputs later
  • input FWHM 4.84+/-0.81 (ave?)
  • target seeing FWHM: 6.88, kron ~6.29
filename FWHM_MAJ (pixel) IQ_FW1 (pixel) .log Input FWHM (pixel) .log kron (pixel)
real_001 x x 5.40 4.39/35
real_009 x x 4.65 3.63/62
fake_001 x x
fake_009 x x
conv.0 x x
conv.9 x x
comb.initial x x
stack 14.36 9.75 (was missing from header)
stack.unconv 4.31 4.15

Sim: 20 input case: 5 groups

Simulated inputs:

  • basic set of 4 images of growing FWHM and repeated as 4 groups for 20 inputs
  • input FWHM 4.57+/-0.72 (mean)
  • target seeing FWHM: 5.70 -- set by worst input, Kron ~3.1 from log
  • some wildly varying kron values
filename sim FWHM sim FWHM_MAJ .log Input FWHM (pixel) .log kron (pixel)
_000 1.0 3.82 etc 3.82(1) 1.99/92,88/78,05/05,91/81
_001 1.1 4.19(1) 2.19/09
_002 1.2 4.58(1) 2.55/48
_003 1.5 5.69(1) 3.11/00

Additional and final outputs:

filename FWHM_MAJ (pixel) IQ_FW1 (pixel)
real_000 3.88 3.00
real_003 5.82 4.54
fake_000 5.10 3.69
fake_003 5.04 3.68
conv.1 6.51 5.10
conv.3 6.60 5.19
comb.initial 6.57 5.58
stack 6.57 5.41
stack.unconv 4.33 3.49

filename descriptions:

  • real_ = input warps
  • fake_ = target image to convolve the real to (bit more than just that, selected target PSF sources, background, scaling)
  • conv. = convolved real image
  • comb.initial = initial stack w/o rejecting input images (that all?)
  • stack = final convolved stack

Sim: 20 input case w/ similar inputs: 10 groups

  • similar set, just created with 2 types with similar FWHM
  • input FWHM 4.07+/-0.19 (ave)
  • target seeing FWHM: 4.28 -- set by worst input, kron ~2.26
filename sim FWHM sim FWHM_MAJ .log Input FWHM (pixel) .log kron (pixel)
_000 1.0 3.87(2) 3.88(1) 2.1(1)
_001 1.1 4.25(1) 4.26(1) 2.28(4)
  • oddly final stack is ~20% larger than conv and initial stack -- need to check -- seems so, may be another issue to watch for (really poor convolved input?)
  • small difference in the fake_and conv. so 20 input case w/ range of inputs better
filename FWHM_MAJ (pixel) IQ_FW1 (pixel)
real_000 3.87 3.00
real_001 4.24 3.32
fake_000 4.27 3.22
fake_001 4.27 3.21
conv.0 4.49 3.52
conv.1 4.56 3.58
comb.initial 4.54 3.77
stack 5.37 4.36
stack.unconv 3.99 3.17

Sim: 100 input case: 25 groups

  • same set as before, just more groups
  • input FWHM 4.56+/-0.71 (mean)
  • target seeing FWHM: 5.72, kron ~3.02
  • an odd run mix-up in this test?
filename sim FWHM sim FWHM_MAJ .log Input FWHM (pixel) .log kron (pixel)
_000 1.0 3.82(1) 1.99/92,88/78,05/05,91/81
_001 1.1 4.19(1) 2.19/09
_002 1.2 4.58(1) 2.55/48
_003 1.5 5.69(1) 3.11/00
  • the x/y in stack is stack.cmf versus running psphot manually on stack, IQ_FW1 is missing from stack.cmf
  • stack.000/stack.0.conv.cmf has 6.26/5.37
  • odd the fake_ target image is much smaller than the target seeing FWHM chosen -- while the convolved images are larger than the target still.
filename FWHM_MAJ (pixel) IQ_FW1 (pixel)
real_000 3.82 2.11/24
real_003 5.69 3.10/02
fake_000 4.86 3.64
fake_003 4.85 3.63
conv.0 6.44 5.74
conv.3 6.61 5.63
comb.initial 8.33 6.89
stack 7.73/8.13 x/6.72
stack.unconv 4.40 3.60

Modification Tests

SYS.ERR/target variance

(lots of previous tracing of stages and such...) SYS.ERR setting history:

For now, setting the SYS.ERR=0 to not mix an N scaled model flux fraction into the variance (SYS.ERR is applied to both stamps)

  • run with current ops tag (20120216), need to run with TESTING turned on or update the branch since there is still some blowup
  • may still want SYS.ERR applied relative to real images, will need to look at deep modification for how/when the variance/weights are passed/calculated
filename sim 20 FWHM_MAJ (pixel) sim 100 FWHM_MAJ (pixel) MD04 20 FWHM_MAJ (pixel) MD04 100 FWHM_MAJ (pixel) MD04 400 FWHM_MAJ (pixel)
input/target FWHM 4.5/5.7 4.6/5.7 4.5/5.3 4.8/6.9 5.1/7.1
real_000
fake_000
conv.0
comb.initial
stack (log/psphot) 5.6/5.8 5.6/5.8 5.8/5.8 7.8/7.8 9.7/9.8
stack.unconv 4.3 4.3 4.0 4.2 4.6

A sys error on the real image may still be useful as a tuning parameter -- sim set of 100

  • initially may have been a mis-run test, re-running shows target~5.7, conv output ~5.6, so no growth
  • selected stamps not significantly different to if sys.err=0 -- better to test on MD04 sample however, need hardcopy plot of stamps chi2 vs flux (TODO)

How does SYS.ERR=0 affect the image rejection list - seems to reject some of the worse seeing images, which are what is used to set the TARGET to begin with. (TODO)

  • May have slightly greater sensitivity for rejecting input images on the worse quality (QUANTIFY).
  • Could also just keep SYS.ERR applied to real image (see above) but may be a different rejection list since before was driven by huge simulated target flux..

Suspect SYS.ERR is also an issue (minor) with the single convolution diffims as well, so add back to test ppSub_testing_201201. Leads into larger development need of splitting/carrying the variances through and not dumping into a weight variable so early (see above).

Need to run larger FPA skycell sample, grizy filters of 100 - 400 to test stability

--> skycell.055 near center
mem400_stack_test.fixtest_s055g/mem400.log:      Input FWHMs : 6.017093 +/- 0.818257              
mem400_stack_test.fixtest_s055i/mem400.log:      Input FWHMs : 5.144256 +/- 0.946591              
mem400_stack_test.fixtest_s055r/mem400.log:      Input FWHMs : 5.681267 +/- 0.910019              
mem400_stack_test.fixtest_s055y/mem400.log:      Input FWHMs : 5.302593 +/- 0.957232              
mem400_stack_test.fixtest_s055z/mem400.log:      Input FWHMs : 5.116535 +/- 0.909322              

mem400_stack_test.fixtest_s055g/mem400.log:      Target seeing FWHM: 10.172999                    
mem400_stack_test.fixtest_s055i/mem400.log:      Target seeing FWHM: 7.128121                     
mem400_stack_test.fixtest_s055r/mem400.log:      Target seeing FWHM: 8.744406                     
mem400_stack_test.fixtest_s055y/mem400.log:      Target seeing FWHM: 8.210619                     
mem400_stack_test.fixtest_s055z/mem400.log:      Target seeing FWHM: 8.907239                     
--> convolved stack PSFs
mem400_stack_test.fixtest_s055g/mem400.log:        fwhm (psf): 10.278124,10.079561 (moments): 0.000000,0.000000
mem400_stack_test.fixtest_s055i/mem400.log:        fwhm (psf): 9.777281,9.695623 (moments): 0.000000,0.000000
mem400_stack_test.fixtest_s055r/mem400.log:        fwhm (psf): 11.805278,11.348278 (moments): 0.000000,0.000000
mem400_stack_test.fixtest_s055y/mem400.log:        fwhm (psf): 8.573956,8.533222 (moments): 0.000000,0.000000
mem400_stack_test.fixtest_s055z/mem400.log:        fwhm (psf): 9.877050,9.810744 (moments): 0.000000,0.000000

--> skycell.058 closer to edge
mem400_stack_test.fixtest_s058g/mem400.log:      Input FWHMs : 5.537649 +/- 0.747769              
mem400_stack_test.fixtest_s058i/mem400.log:      Input FWHMs : 5.045243 +/- 0.928610              
mem400_stack_test.fixtest_s058r/mem400.log:      Input FWHMs : 5.281566 +/- 0.931646              
mem400_stack_test.fixtest_s058y/mem400.log:      Input FWHMs : 5.341974 +/- 1.066415              
mem400_stack_test.fixtest_s058z/mem400.log:      Input FWHMs : 5.282053 +/- 0.859332              

mem400_stack_test.fixtest_s058g/mem400.log:      Target seeing FWHM: 8.574877                     
mem400_stack_test.fixtest_s058i/mem400.log:      Target seeing FWHM: 8.097901                     
mem400_stack_test.fixtest_s058r/mem400.log:      Target seeing FWHM: 7.353693                     
mem400_stack_test.fixtest_s058y/mem400.log:      Target seeing FWHM: 8.535203                     
mem400_stack_test.fixtest_s058z/mem400.log:      Target seeing FWHM: 8.813280                     
--> convolved stack PSFs
mem400_stack_test.fixtest_s058g/mem400.log:        fwhm (psf): 8.743460,8.612661 (moments): 0.000000,0.000000
mem400_stack_test.fixtest_s058i/mem400.log:        fwhm (psf): 9.086349,9.011602 (moments): 0.000000,0.000000
mem400_stack_test.fixtest_s058r/mem400.log:        fwhm (psf): 7.497458,7.397744 (moments): 0.000000,0.000000
mem400_stack_test.fixtest_s058y/mem400.log:        fwhm (psf): 8.861631,8.803410 (moments): 0.000000,0.000000
mem400_stack_test.fixtest_s058z/mem400.log:        fwhm (psf): 9.594481,9.520103 (moments): 0.000000,0.000000

Seems to be a few issues in this larger sample -

  • if FWHM cut >7.5 pixels, why is target selected >7.5? Are input images not being rejected? Somewhat part of next section on Target PSF, but need to trace out here. Could split up 400 set into subgroups and run. Looks like rejected are not being used but is growing itself in converting to the PSF.MODEL PS_MODEL_PS1_V1 or just in the envelope itself to include some wacky IQ images.
  • s055g - rerun in smaller sub-sets ~100, target is ~9.45 and has similar output. Limiting input FWHM <7.1 selects a target ~8.4. rejection for >7.5 is active. The large target PSF looks to be trying to account for inputs with odd IQ in the envelope calculation and will need to collect stats on the input images for comparison.
  • s055r - appears to have a problem, the convolved images are poor and many are rejected. Target PSF is in some way not representative? 100 input subsets seem to not show this problem. Wacky bug, see pmReadoutFake footprint/stamp radius typecast to not 0.0 in threading.
  • s055i - largest growth still from target to final stack (~2 pixels). The target image PSF is ~7 pixels as says, however all the convolved input images are 9-11 pixels. Rerunning this and see if can reproduce, seems so in the pmReadoutFake footprint/stamp radius typecast to not 0.0 in threading. Running sub-groups ~100 doesn't have this problem.
  • generally would be helpful to have other stats, at least min/max inputs on the envelope/target PSF calculation.

Related to the s055r problem, there may be a problem with excess image rejection: generally ~20-40% are from the >7.5 pixel cut initially

  • Note -- rejected will be cut for input into the unconvolved stacks as well. Even for 058g,i seems a bit higher than would want.
    mem400_stack_test.fixtest_s055g/mem400.stats:REJECT_IMAGES    S32       115              # Number of images rejected completely
    mem400_stack_test.fixtest_s055i/mem400.stats:REJECT_IMAGES    S32       89               # Number of images rejected completely
    mem400_stack_test.fixtest_s055r/mem400.stats:REJECT_IMAGES    S32       318              # Number of images rejected completely
    mem400_stack_test.fixtest_s055r_redo/mem400.stats:REJECT_IMAGES    S32       317              # Number of images rejected completely
    mem400_stack_test.fixtest_s055y/mem400.stats:REJECT_IMAGES    S32       45               # Number of images rejected completely
    mem400_stack_test.fixtest_s055z/mem400.stats:REJECT_IMAGES    S32       45               # Number of images rejected completely
    
    mem400_stack_test.fixtest_s058g/mem400.stats:REJECT_IMAGES    S32       101              # Number of images rejected completely
    mem400_stack_test.fixtest_s058i/mem400.stats:REJECT_IMAGES    S32       122              # Number of images rejected completely
    mem400_stack_test.fixtest_s058r/mem400.stats:REJECT_IMAGES    S32       54               # Number of images rejected completely
    mem400_stack_test.fixtest_s058y/mem400.stats:REJECT_IMAGES    S32       61               # Number of images rejected completely
    mem400_stack_test.fixtest_s058z/mem400.stats:REJECT_IMAGES    S32       62               # Number of images rejected completely
    
  • see pmReadoutFake footprint/stamp radius typecast to not 0.0 in threading

pmReadoutFake footprint/stamp radius typecast fixed not 0.0 in threading

Re-run MD04@400 for s055 grizy including the SYS.ERR=0 and typecast fix:

mem400_stack_test.fixtest_s055g/mem400.log:      Target seeing FWHM: 10.172999                                                          
mem400_stack_test.fixtest_s055i/mem400.log:      Target seeing FWHM: 7.128121                                                           
mem400_stack_test.fixtest_s055r/mem400.log:      Target seeing FWHM: 8.744406                                                           
mem400_stack_test.fixtest_s055y/mem400.log:      Target seeing FWHM: 8.210619                                                           
mem400_stack_test.fixtest_s055z/mem400.log:      Target seeing FWHM: 8.907239                                                          

mem400_stack_test.fixtest_s055g/mem400.log:        fwhm (psf): 10.245927,10.035181 (moments): 0.000000,0.000000                         
mem400_stack_test.fixtest_s055i/mem400.log:        fwhm (psf): 9.053534,9.025082 (moments): 0.000000,0.000000                           
mem400_stack_test.fixtest_s055r/mem400.log:        fwhm (psf): 9.291595,9.220863 (moments): 0.000000,0.000000                           
mem400_stack_test.fixtest_s055y/mem400.log:        fwhm (psf): 8.697776,8.658995 (moments): 0.000000,0.000000                           
mem400_stack_test.fixtest_s055z/mem400.log:        fwhm (psf): 9.827286,9.793766 (moments): 0.000000,0.000000         

mem400_stack_test.fixtest_s055g/mem400.stats:REJECT_IMAGES    S32       115              # Number of images rejected completely         
mem400_stack_test.fixtest_s055i/mem400.stats:REJECT_IMAGES    S32       88               # Number of images rejected completely         
mem400_stack_test.fixtest_s055r/mem400.stats:REJECT_IMAGES    S32       71               # Number of images rejected completely         
mem400_stack_test.fixtest_s055y/mem400.stats:REJECT_IMAGES    S32       50               # Number of images rejected completely         
mem400_stack_test.fixtest_s055z/mem400.stats:REJECT_IMAGES    S32       43               # Number of images rejected completely         

Fixes r-band issue, others mostly unaffected. Could affect other code/stages tuned params, need to check before adding to trunk+tag -- nothing drastic in samples so added fix.

Looks to be now a question of tracing the target PSF selection process for the g-band and the extra growth on i-band. Becoming an additional branch of investigation related to the IQ of the inputs going into the stacks that the PSF envelope solution is made from and need to trace down to individual inputs.

Break up into groups, then break into smaller sub-sets the groups having issues.

s055g: Target and convolved output PSF are relatively well matched except much greater than other target PSFs. Why.

  • appears the normal maximum input FWHM driver are being rejected, ~80/400 in fact..
    ...
          Input 186: 6.528622
          Input 187: 11.727304
          Input 188: 11.179210
    ...
          PSF FWHM for image 187 is too large (11.727304 vs 7.500000 maxFWHM) --- rejected.
          PSF FWHM for image 188 is too large (11.179210 vs 7.500000 maxFWHM) --- rejected.
    ...
    
  • then likely the PSF envelope trying to fit some odd inputs being an edge skycell, and break into groups to run - ~100 inputs
    • 0-100 group okay, Target PSF ~6.7 pixels
    • 100-200 group has issue with Target PSF ~10 -- something in this group
            Input FWHMs : 6.206581 +/- 0.880895
            Target seeing FWHM: 9.458570
      
    • subdivide further -- s2s1 for ~25 - drop the 7-7.5 ones - 15, 19-25,49,51,54-55,58-59,61-62 (remember to add +1) -
            Input FWHMs : 5.867408 +/- 0.759102
            Target seeing FWHM: 8.387546
      --> so it is the conversion of 7.5-> PS1 PSF? except full g gets to ~10.1, and below 7.3 gets to 8.3 so factor somewhere else still
      --> so could make set of high 7.x -- use the s2 set and add only -- s2s2 - 15, 19-25,49,51,54-55,58-59,61-62 - add +1 to list 
            Input 0: 7.496574
            Input 1: 7.430977
            Input 2: 7.192523
            Input 3: 7.257564
            Input 4: 7.222930
            Input 5: 7.049495
            Input 6: 7.181604
            Input 7: 6.598584
            Input 8: 5.182988
            Input 9: 7.168517
            Input 10: 7.095002
            Input 11: 7.000916
            Input 12: 7.146842
            Input 13: 7.231784
            Input 14: 7.243643
            Input 15: 7.007473
            Input 16: 7.409586
            Input FWHMs : 7.170876 +/- 0.208919
                fit ext:   1.898029 sec for 225 of 225 sources                                                         |
                fit psf:   0.426171 sec for 161 of 225 sources                                                         |
              apresid: 0.120900 +/- 0.005411 (systematic error) from statistics of 225 psf stars (161 used)
              chisq vs flux fit term 0: 0.000000 +/- 0.000000
              chisq vs flux fit term 1: 0.000000 +/- 0.000000
            try model PS_MODEL_PS1_V1, ap-fit: 0.120900 +/- 0.005411 : sky bias: 0.000000
            Target seeing FWHM: 9.458885                                                                      
      --> so looks like 7.49->9.45
      
    • s2s3 for <7.4 and still 9.45 -- also 7.2, only to 7.1
    • s2s4 - clear down to 7.1 -- drop 0,1,2,4,11,12,14
            Input 0: 7.049495
            Input 1: 6.598584
            Input 2: 5.182988
            Input 3: 7.095002
            Input 4: 7.000916
            Input 5: 7.007473
            Input FWHMs : 6.655743 +/- 0.743387
                fit ext:   1.873564 sec for 225 of 225 sources                                                         |
                fit psf:   0.475994 sec for 172 of 225 sources                                                         |
              apresid: 0.102933 +/- 0.013974 (systematic error) from statistics of 225 psf stars (172 used)
              chisq vs flux fit term 0: 0.000000 +/- 0.000000
              chisq vs flux fit term 1: 0.000000 +/- 0.000000
            try model PS_MODEL_PS1_V1, ap-fit: 0.102933 +/- 0.013974 : sky bias: 0.000000
            Target seeing FWHM: 8.786390                                  
      
    • 200-345 group also issue ~10
    • 300-345? yes ~8.3 -- s2s4 is test case then - try to track down which they are -
    • s41 ~7.6 -- how many in/rejected?
    • s42 ~6.8 but highest <7.5 is 6.2 and only couple
    • s43 ~6.4 and clear
    • s44 ~8.35 in/rej~14/6, 7.3 largest -- use this one, lots of large
      • so then which ones in this set are causing the large PSF and why? maybe not too unlikely that it would change 7.3->8.3 so do another above
    • specific sub test -- mem20_stack_test.fixfwhmincut055g
            PSF FWHM for image 1 is too large (7.785680 vs 7.500000 maxFWHM) --- rejected.
            PSF FWHM for image 5 is too large (7.621037 vs 7.500000 maxFWHM) --- rejected.
            PSF FWHM for image 9 is too large (7.927819 vs 7.500000 maxFWHM) --- rejected.
            PSF FWHM for image 11 is too large (7.701270 vs 7.500000 maxFWHM) --- rejected.
            PSF FWHM for image 12 is too large (9.293368 vs 7.500000 maxFWHM) --- rejected.
            PSF FWHM for image 13 is too large (9.460286 vs 7.500000 maxFWHM) --- rejected.
            Input seeing FWHMs:
            Input 0: 7.312490
      --      Input 1: 7.785680
            Input 2: 6.733018
            Input 3: 7.106098
            Input 4: 7.107179
      --      Input 5: 7.621037
            Input 6: 6.591544
            Input 7: 7.348337
            Input 8: 6.802173
      --      Input 9: 7.927819
            Input 10: 7.252710
      --      Input 11: 7.701270
      --      Input 12: 9.293368
      --      Input 13: 9.460286
            Input FWHMs : 7.031693 +/- 0.286558
                fit ext:   1.920824 sec for 225 of 225 sources
                fit psf:   0.459095 sec for 168 of 225 sources
              apresid: 0.102416 +/- 0.002603 (systematic error) from statistics of 225 psf stars (168 used)
              chisq vs flux fit term 0: 0.000000 +/- 0.000000
              chisq vs flux fit term 1: 0.000000 +/- 0.000000
            try model PS_MODEL_PS1_V1, ap-fit: 0.102416 +/- 0.002603 : sky bias: 0.000000
            Target seeing FWHM: 8.350183
      
  • may indicate only able to happen with large numbers of varying IQ inputs, not with 20
  • really need more output details from target PSF selection to know what it is doing..
  • -- OOPS -- unfortunately these were cleaned before listings saved so cannot trace filenames w/o rerunning...

s055i: While the FWHM~7.1, the convolved images have FWHM~9-10 generally.

FILENAME                                         FWHM_MAJ IQ_FW1
mem400_deepstack.ppstack.0.conv.im.pht.fpa.cmf   10.47247 8.178617
mem400_deepstack.ppstack.1.conv.im.pht.fpa.cmf   11.10159 8.916221
mem400_deepstack.ppstack.10.conv.im.pht.fpa.cmf  9.585663 8.479208
mem400_deepstack.ppstack.100.conv.im.pht.fpa.cmf 10.89854 8.851029
mem400_deepstack.ppstack.101.conv.im.pht.fpa.cmf 10.68771 8.692747
mem400_deepstack.ppstack.102.conv.im.pht.fpa.cmf 10.64534 8.854232
mem400_deepstack.ppstack.103.conv.im.pht.fpa.cmf 9.786881 9.003357
mem400_deepstack.ppstack.104.conv.im.pht.fpa.cmf 10.20106 8.508514
mem400_deepstack.ppstack.105.conv.im.pht.fpa.cmf 10.36447 8.872278
mem400_deepstack.ppstack.106.conv.im.pht.fpa.cmf 11.17089 8.858175
mem400_deepstack.ppstack.107.conv.im.pht.fpa.cmf 11.02701 9.404055
mem400_deepstack.ppstack.108.conv.im.pht.fpa.cmf 10.61196 8.608229
mem400_deepstack.ppstack.109.conv.im.pht.fpa.cmf 11.64145 8.561131
mem400_deepstack.ppstack.11.conv.im.pht.fpa.cmf  10.83251 8.612485
  • Break into sub-groups of different sizes (default is ~100 inputs)
    mem400_stack_test.fixtest_s055i_s1sub/mem400.log:      Target seeing FWHM: 6.948271
    mem400_stack_test.fixtest_s055i_s2sub/mem400.log:      Target seeing FWHM: 7.983406
    mem400_stack_test.fixtest_s055i_s3sub/mem400.log:      Target seeing FWHM: 8.228301
    mem400_stack_test.fixtest_s055i_s4sub/mem400.log:      Target seeing FWHM: 7.010231
    mem400_stack_test.fixtest_s055i_s1sub/mem400.log:        fwhm (psf): 7.771870,7.675667 (moments): 0.000000,0.000000
    mem400_stack_test.fixtest_s055i_s2sub/mem400.log:        fwhm (psf): 8.531821,8.457088 (moments): 0.000000,0.000000
    mem400_stack_test.fixtest_s055i_s3sub/mem400.log:        fwhm (psf): 8.927197,8.827012 (moments): 0.000000,0.000000
    mem400_stack_test.fixtest_s055i_s4sub/mem400.log:        fwhm (psf): 8.465529,8.424333 (moments): 0.000000,0.000000
    
    mem400_stack_test.fixtest_s055i_s1s200sub/mem400.log:      Target seeing FWHM: 7.993505
    mem400_stack_test.fixtest_s055i_s1s300sub/mem400.log:      Target seeing FWHM: 8.162872
    mem400_stack_test.fixtest_s055i_s1s400sub/mem400.log:      Target seeing FWHM: 7.128121
    mem400_stack_test.fixtest_s055i_s1s200sub/mem400.log:        fwhm (psf): 8.951500,8.863860 (moments): 0.000000,0.000000
    mem400_stack_test.fixtest_s055i_s1s300sub/mem400.log:        fwhm (psf): 9.282644,9.198133 (moments): 0.000000,0.000000
    mem400_stack_test.fixtest_s055i_s1s400sub/mem400.log:        fwhm (psf): 9.053215,9.020257 (moments): 0.000000,0.000000
    
    mem400_stack_test.fixtest_s055i_s2s200sub/mem400.log:      Target seeing FWHM: 7.011785
    mem400_stack_test.fixtest_s055i_s2s300sub/mem400.log:      Target seeing FWHM: 7.123811
    mem400_stack_test.fixtest_s055i_s2s200sub/mem400.log:        fwhm (psf): 8.598989,8.531817 (moments): 0.000000,0.000000
    mem400_stack_test.fixtest_s055i_s2s300sub/mem400.log:        fwhm (psf): 8.875980,8.798958 (moments): 0.000000,0.000000
    
    • Appears to choose a much better FWHM for more recent data (s4) but overlap of s2s200 versus what a larger FWHM for s3 shows indicates something is going strange in s4.
  • Break sub-groups into sub-sets
    mem400_stack_test.fixtest_s055i_s4s1sub/mem400.log:      Target seeing FWHM: 7.462551
    mem400_stack_test.fixtest_s055i_s4s2sub/mem400.log:      Target seeing FWHM: 5.529995
    mem400_stack_test.fixtest_s055i_s4s3sub/mem400.log:      Target seeing FWHM: 6.343721
    mem400_stack_test.fixtest_s055i_s4s4sub/mem400.log:      Target seeing FWHM: 7.060447
    
    mem400_stack_test.fixtest_s055i_s4s1sub/mem400.log:        fwhm (psf): 7.691631,7.642853 (moments): 0.000000,0.000000
    mem400_stack_test.fixtest_s055i_s4s2sub/mem400.log:        fwhm (psf): 5.841974,5.706679 (moments): 0.000000,0.000000
    mem400_stack_test.fixtest_s055i_s4s3sub/mem400.log:        fwhm (psf): 6.399483,6.340310 (moments): 0.000000,0.000000
    mem400_stack_test.fixtest_s055i_s4s4sub/mem400.log:        fwhm (psf): 7.731044,7.698786 (moments): 0.000000,0.000000
    --> so s4s5.. and some s3s3-4 inputs may be involved?
    mem400_stack_test.fixtest_s055i_s4s5sub/mem400.log:      Target seeing FWHM: 6.130914
    mem400_stack_test.fixtest_s055i_s4s5sub/mem400.log:        fwhm (psf): 6.409942,6.297189 (moments): 0.000000,0.000000
    
    mem400_stack_test.fixtest_s055i_s3s3sub/mem400.log:      Target seeing FWHM: 7.966679
    mem400_stack_test.fixtest_s055i_s3s4sub/mem400.log:      Target seeing FWHM: 7.331856
    mem400_stack_test.fixtest_s055i_s3s3sub/mem400.log:        fwhm (psf): 8.106666,7.973850 (moments): 0.000000,0.000000
    mem400_stack_test.fixtest_s055i_s3s4sub/mem400.log:        fwhm (psf): 7.526990,7.406355 (moments): 0.000000,0.000000
    
  • Manually rerun without selected input/exposures:
    neb://@HOST@.0/gpc1/condor_MD04.V3_01.haf/o5186g0067o.113047/o5186g0067o.113047.wrp.276921.skycell.055.fits -- fw~1.9
    --> Target seeing FWHM drops to ~7.1 (from 8.2)
    neb://@HOST@.0/gpc1/condor_MD04.V3_01.haf/o5186g0072o.113052/o5186g0072o.113052.wrp.276963.skycell.055.fits -- fw~1.7
    neb://@HOST@.0/gpc1/condor_MD04.V3_01.haf/o5186g0060o.113040/o5186g0060o.113040.wrp.276911.skycell.055.fits
    --> Target seeing FWHM drop to ~8.0  
    
    • seems a single odd exposure can drive the target PSF strangely where FWHM~7.1 but sets the convolved images ~9-10.
    • o5186 exposures all help push the Target seeing FWHM down when included with larger sample of others.
    • not a case of extra growth of the Target to output PSF, but odd value of the Target PSF.

Really looking now issue of types of inputs that drive the target PSF to "strange" values (typically larger for the PSF envelope to match, but also smaller). Can specific cuts be used to restrict odd or largely deviant inputs for the target PSF?

  • really an external approach to cut inputs versus internal rejection from large deconvolution likely when using modified Target PSF, so hold here until work through Target PSF choice.

Larger sample test with skycell.02x ipp-20120404 tag (4/9/12)

Rerun MD04@all inputs using the ipp-20120404 tag with SYS.ERR=0, stamp size typecast fix, and minor others (need to note date since various changes been happening in tag -- 4/9/12)

mysql> select stack_id,state,path_base,skycell_id from stackRun join stackSumSkyfile using (stack_id) where data_group like "%deeptest.2012%";
+----------+-------+------------------------------------------------------------------------------------------+-------------+
| stack_id | state | path_base                                                                                | skycell_id  |
+----------+-------+------------------------------------------------------------------------------------------+-------------+
|   848697 | new   | neb://any/gpc1/MD04.deeptest.20120406/MD04.V3/skycell.021/MD04.V3.skycell.021.stk.848697 | skycell.021 | 
|   848698 | full  | neb://any/gpc1/MD04.deeptest.20120406/MD04.V3/skycell.022/MD04.V3.skycell.022.stk.848698 | skycell.022 | 
|   848699 | full  | neb://any/gpc1/MD04.deeptest.20120406/MD04.V3/skycell.023/MD04.V3.skycell.023.stk.848699 | skycell.023 | 
|   848700 | full  | neb://any/gpc1/MD04.deeptest.20120406/MD04.V3/skycell.024/MD04.V3.skycell.024.stk.848700 | skycell.024 | 
|   848701 | full  | neb://any/gpc1/MD04.deeptest.20120406/MD04.V3/skycell.025/MD04.V3.skycell.025.stk.848701 | skycell.025 | 
|   848702 | full  | neb://any/gpc1/MD04.deeptest.20120406/MD04.V3/skycell.026/MD04.V3.skycell.026.stk.848702 | skycell.026 | 
|   848703 | full  | neb://any/gpc1/MD04.deeptest.20120406/MD04.V3/skycell.027/MD04.V3.skycell.027.stk.848703 | skycell.027 | 
|   848704 | full  | neb://any/gpc1/MD04.deeptest.20120406/MD04.V3/skycell.028/MD04.V3.skycell.028.stk.848704 | skycell.028 | 

Run a bit more restrictive then before with an external cap at FWHM major<7.5 from database/camera exposure

stacktool -definebyquery -dbname gpc1 -set_dist_group NULL -set_data_group MD04.deeptest.20120406 -set_label MD04.deeptest.20120406 -set_workdir neb://@HOST@.0/gpc1/MD04.deeptest.20120406 -select_data_group condor_MD04.V3_01_dg -select_filter g.00000 -select_good_frac_min 0.05 -min_num 2 -select_fwhm_major_max 7.5 -set_reduction DEEP_STACK -simple -select_skycell_id skycell.02%  -pretend >& run_stacktool.g.md04_deeptest.20120406.fwhm75_n2.log
Target seeing g r i
skycell.021 fault fault fault
022 8.89 7.94 7.67
023 8.25 8.30 9.02
024 8.21 7.74 7.36
025 8.30 7.91 7.88
026 8.44 8.76 7.63
027 7.75 9.06 9.43
028 15.80 8.65 9.61
029 8.50 9.42 7.98

As expected, see possible extreme target PSF with poorly constrained input quality and not just on the edges.

Sample run highlighted another issue with skycell.021 fault due to not enough sources match for transparency measurements that then triggers a full fault. Odd because there are sources in the field even though in a corner. What sets the minimum number of sources?

  • ppStack log notes "Selected 0 sources for photometry analysis"
  • ppStackSources.c -- modified section options->photometry where minMatches should be in reference to numGoodImages and not the full num images since if many are rejected (near edge) then becomes useless overly conservative match fraction.

skycell.021-g: another issue revealed where many transparency values are NAN but still uses the FWHM for the Target PSF because of order in the code.. really want to do all possible "cuts" before setting Target PSF.

      ...
      Input 293: 7.673504
      Input FWHMs : 6.015619 +/- 0.671127
    PSF 240 is completely bad --- not including in envelope calculation.
     ...
      Target seeing FWHM: 9.178185

--> final Ninput=7 (tedious to trace out) and undershot the target PSF value, not sure how.
        fwhm (psf): 7.569923,7.118117 (moments): 0.000000,0.000000

Change made in ppStackPrepare.c -- move ppStackSourcesTransparency before generate target PSF/ppStackPSF section.

  • Note -- this will invalidate the TESTING section that produces the start_xxx.fits images in ppStackSourcesTransparency that uses the defined PSF.TARGET

Target PSF for Convolved Stacks

Want to explore not necessarily setting the convolved "matched/equilized/homogenized" deep-stack target PSF to what generally appears to be driven by the worst input. Have seen cases when the poor inputs could be image rejected and in that case not a good idea to use anyways, and the target PSF re-chosen and stack remade. How can poor imputs be pre-filtered better? Is a most common input based target PSF better? What level of deconvolution is acceptable?

  • simple cuts:
    • for large input N, a rerun of PSF evelope/target PSF would be of little time cost but rerun of all the image convolutions is terribly expensive.
    • for small input N, enter into decision of adding poorer images versus dropping 20-30% of possible inputs.
  • instead of a re-run of target PSF with additional image rejection, could choose mean/median PSF or an optimal and use poor deconvolution to reject worse images?
  • important to remember, rejecting inputs for convolved stacks is the same for unconvolved stacks (for the pixel rejection method), so excessive rejection not good for either.
  • expect the rejecting and target PSF cases to possibly be different for MD (similar good inputs, easy to clip worse seeing tail without significant loss of depth) versus LAP (similar to wildly variable quality inputs, maybe few far outliers or maybe even spread of quality levels).

Purpose of convolved stacks?

  • staticsky photometry -- convolved stacks have a similar PSF across the field for photometry and morphology measurements. However, also then goes through another round of (de)convolution/smoothing to match the PSF of all input filters.
  • matched PSF over skycell, semi-matched PSF over larger areas? How necessary is that currently?
  • specific science reasons for either or?

Test cases:

  • Case 0: Greatly reduce the input image FWHM limits and accept possibly fewer inputs into the stacks. For stacks that fail without enough inputs, have a second pass with more relaxed limits. Pass 1 could also require a larger number of minimum inputs (larger than the normal 2-6 inputs currently) with a very restrictive "FWHM" cut (good). Pass 2 more relaxed allowing fewer minimum inputs and/or relaxed cuts (good as going to get).
    • affects the unconvolved and well as the convolved however (though 3" seeing images isn't helping either way)
  • Case 1: Enhance the already exising simple model target PSF code to interally use the already calculated input FWHM mean (and stdev) to set a simple target PSF.
    • is a convolution to a simple model PSF workable
    • how much deconvolution is acceptable in the inputs to the stack
  • Case 2: Either modify the PSF envelope code to trend towards the more likely target PSF or sub-select the inputs for the the target PSF to be set to.
    • open question if the PSF envelope code is behaving as intendent and needs to be tested further
    • then similar to case 1 deconvolution acceptable over the field

Test runs (same as defined above ppStack_testing_201111#TestSets):

  • all previous improvements included, like SYS.ERR=0, comes after this stage for the convolution mainly so not important for picking target, but is important for final photometry/property comparisons.
  • since comparing between different runs, will want to use same SEED values for ppStack and psphot at least initially. Will also want exactly same input warps copies (i.e., from simtest run and not regenerate for each to avoid any differences even if using same seed there as well).
  • be sure to note PSF model choices for the two samples, will want to try match initially, but also look at when not well matched later.
  • what is the baseline to use for relative comparison? Unconvolved stacks are made at same time, but can depend on the comparison. Simulated set has known values for its creation.
  • will want to include spatial variations, simtest sample should be uniform but MD04 will not be and may have "strange" inputs to note and make use of.
  • not sure how to included information rejected images by some IQ or deconvolution limits.
  • Simtest sample:
    • may want to make a sample distribution of simulated PSF FWHMs, currently sample is flat. Would need to revise the making of the stack.mdc list of inputs, or will just be worse/upper limit of possible stack degrading for including deconvolved images.
    • start with sample N=20, not clear what N=100 would help to do here.
    • PSF model is _GAUSS for simulated images, start with matched target model and mpdel in psphot. Will want to look at different matched target models.
  • MD04 sample:
    • for sub-samples of skycell.055: N=20 fairly irregular sample, N=100 similar to full sample
    • skycell.055,058 for full test samples for deep stacks: current operating FWHM cut is at 7.5 pixels, generally excluding <10% of all available warps.
  • PSF model choices?

Case 0: Improved Cuts/Limits

Possible preliminary improvement: really can only judge with real data samples Sample summary of past LAP run stacks ~mhuber/lapstack20120510_inputtargetpsffwhm.log

  • summary plots tedious but would be helpful to visualize how ofter more restrictive cuts would help

past/current ppStack.config for STACK_THREEPI:

  • PSF.INPUT.MAX 12.0 -- anything >12 pixels excluded (3")
  • PSF.INPUT.CLIP.NSIGMA NAN -- not used, but would redefine the INPUT.MAX to be clipped_mean + NSIGMA*clipped_stdev

proposed:

  • PSF.INPUT.MAX 8.0 (or 2", MD deep stack is set for 7.5)
  • PSF.INPUT.CLIP.NSIGMA 2.0 (or 1.0?)

Nightly stacks (primarily MD concerned) have NAN for PSF.INPUT.MAX and so is not set. If the night is decent, should be similar and minor issue (should get stats on though TBT)

Chris quickly looked at the FWHM of the exposures going into LAP stacks and order 90 percentile are <8 pixels. The NSIGMA is more difficult to quantify, so the general question is do we want to just help trim off more of the poorest inputs to keep the convolved target PSF more regular or be much more restrictive to improve the convolved target PSF at the cost of the number of inputs (maybe 10-30% at times for NSIGMA of 1). Any additional exclusion cuts include images that go into making the unconvolved stack as well.

Adding pass cases for limits on the different minimum number of inputs would require some task modifications in LAP (and nightly science).

Should add samples of N inputs with/without the worst inputs for contribution in unconvolved stacks.

Case 1: Initial Manually Set Target as Simple Model PSF (include FWHM)

To explore the implications of (de)convolving to a simple PSF model with simtest and MD data

Key config options:

  • recipes/ppStack.config
    PSF.MODEL       STR     PS_MODEL_GAUSS  # Model for PSF generation
    
    PSF.AUTOSIZE            BOOL    FALSE           # Determine output PSF size from input PSFs?
    PSF.OUTPUT.FWHM         F32     5.0             # Target size for output PSF (if not auto-sized), -1 as FWHM
    PSF.OUTPUT.NFWHMSIG     F32     0.0             # N FWHM-sig to be added to the FWHMmean for PSF
    -- simtest also has a ppStack.config -- be aware of DECONV.LIMIT, watch rejections
    PSF.MODEL       STR     PS_MODEL_GAUSS  # Model for PSF generation
    DECONV.LIMIT    F32     1.5             # Deconvolution fraction for rejecting entire image
    IMAGE.REJ       F32     0.3             # Rejected pixel fraction threshold for rejecting entire image
    
  • recipes/psphot.config
    PSF_MODEL                           STR   PS_MODEL_GAUSS
    EXT_MODEL                           STR   PS_MODEL_GAUSS
    CONSTANT_PHOTOMETRIC_WEIGHTS        BOOL  FALSE 
    POISSON.ERRORS.PHOT.LMM             BOOL  TRUE 
    EXTENDED_SOURCE_FITS                BOOL  FALSE 
    EXTENDED_SOURCE_ANALYSIS            BOOL  FALSE
    -- recipes/psphot.config STACKPHOT uses -- so will probably want to change on cmdline
    PSF_MODEL                           STR   PS_MODEL_PS1_V1 
    EXT_MODEL                           STR   PS_MODEL_QGAUSS
    CONSTANT_PHOTOMETRIC_WEIGHTS        BOOL  TRUE 
    POISSON.ERRORS.PHOT.LMM             BOOL  TRUE
    -- simtest has a psphot.config
    PSF_MODEL                           STR   PS_MODEL_GAUSS
    PSF_CLUMP_GRID_SCALE                F32   1.0             # size of Sx,Sy image pixel
    
    

Basic Script:

  • ppstack seed=16587045608812410537
  • psphot seed=2818454766199170413
  • -Db PSF.AUTOSIZE <X> -Df PSF.OUTPUT.FWHM <X>: [T,F,F,F,F,F,F,F], [0.0, 3.8, 4.2, 4.6, 5.0, 5.4, 5.7, -1.0]
    ppStack -threads 4 -input stack.000/stack.mdc stack.000/stack -Db TEMP.DELETE F -R PPSTACK.OUTPUT.VARIANCE FITS.TYPE NONE -log test.log -recipe PPSTACK STACK_DEEP -recipe PPSUB STACK -recipe PSPHOT STACK -recipe PPSTATS STACKSTATS -stats test.stats -dumpconfig test.mdc -tracedest test.trace -trace err 10 -trace ppStack 10 -trace ppSub 10 -trace psModules.imcombine 10  -trace psModules.camera 10 -vvv -debug -Db PSF.AUTOSIZE F -Df PSF.OUTPUT.FWHM -1.0 -seed 16587045608812410537  >& run_stackdeep.log 
    
    ## keep copy of local convolved input warps  
    mkdir tmpconv
    cp /local/ipp/tmp/stack.*fits tmpconv/
    
    ## normal psphot + STACKPHOT
    psphot -file stack.000/stack.fits -mask stack.000/stack.mask.fits -variance stack.000/stack.weight.fits stack.fullpht -dumpconfig fullpht_test.mdc -seed 2818454766199170413 -threads 4 -log fullpht_test.log
    psphot -file stack.000/stack.unconv.fits -mask stack.000/stack.unconv.mask.fits -variance stack.000/stack.unconv.wt.fits stack.unconv.fullpht  -seed 2818454766199170413 -threads 4 -log fullpht_unconvtest.log
    
    psphot -file stack.000/stack.fits -mask stack.000/stack.mask.fits -variance stack.000/stack.weight.fits stack.stkpht -dumpconfig stkpht_test.mdc -recipe PSPHOT STACKPHOT -D PSF_MODEL PS_MODEL_GAUSS -D EXT_MODEL PS_MODEL_GAUSS -seed 2818454766199170413 -threads 4  -log stkpht_test.log
    psphot -file stack.000/stack.unconv.fits -mask stack.000/stack.unconv.mask.fits -variance stack.000/stack.unconv.wt.fits stack.unconv.stkpht -recipe PSPHOT STACKPHOT -D PSF_MODEL PS_MODEL_GAUSS -D EXT_MODEL PS_MODEL_GAUSS -seed 2818454766199170413 -threads 4 -log stkpht_unconvtest.log
    
    ## (de)convolved input warps
    psphot -file tmpconv/stack.0.conv.im.fits -mask tmpconv/stack.0.conv.mk.fits -variance tmpconv/stack.0.conv.var.fits stack.0.conv.fullpht -dumpconfig tmpconvstack.0.conv.fullpht.mdc -seed 2818454766199170413 -threads 4 -log fullpht.0.conv_test.log
    ..(1,2,3)..
    
    ## input warps -- should always be same, but should run to keep settings up to date
    psphot -file image.000/image.0.wrp.fits -mask image.000/image.0.wrp.mask.fits -variance image.000/image.0.wrp.wt.fits image.0.wrp.fullpht -seed 2818454766199170413 -dumpconfig image.0.wrp.fullpht.mdc -threads 4 -log fullpht.0.wrp_test.log
    ..(1,2,3)..
    
    ## header stats to log (easier to read)
    /home/panstarrs/mhuber/bin/gethead -auh *.pht*.cmf FWHM_MAJ IQ_FW1
    
    

NOTE: if wrongly called on command line with a bool -Db PSF.OUTPUT.FWHM 5.0 will silently fail and use the default configuration setting in ppStack.config

Sim set: currently on disk /data/ipp060.0/mhuber

  • previous version (r?) in testppstack_deep201111/simstest/simdeep120321
    • (need to add details later)
  • revised version (6/21) moved to testppstack_deep201111/simstest/simdeep120619/prerun120621
    • mostly similar run to revised (6/28), sample plots in main directory and of name format test.<A>.<B>.<page#>.png for delta plots <A>-<B> values (need to add indication which is delta and which is normal)
  • revised version (6/28) run summary directories -- plot format similar
    • ppstack_simdeep20.testpsfenvel -- normal PSF envelope version (from sim20 set, target PSF should be ~5.7 pixels, ~1.5" or largest input)
    • ppstack_simdeep20.testsetpsf10 -- selected target PSF to be 1.0" (3.8 sim pixels)
    • ppstack_simdeep20.testsetpsf11 -- selected target PSF to be 1.1" (4.2 sim pixels)
    • ppstack_simdeep20.testsetpsf12 -- selected target PSF to be 1.2" (4.6 sim pixels)
    • ppstack_simdeep20.testsetpsf13 -- selected target PSF to be 1.3" (5.0 sim pixels)
    • ppstack_simdeep20.testsetpsf14 -- selected target PSF to be 1.4" (5.4 sim pixels)
    • ppstack_simdeep20.testsetpsf15 -- selected target PSF to be 1.5" (5.7 sim pixels)
    • ppstack_simdeep20.testsetpsffwhm -- simple PSF model sized to FWHM of inputs (for default sim20 set, should be ~4.5 pixels, 1.2")
    • ppstack_simdeep20.testpsfjunk -- as junk name indicates, junk or misc test runs
    • summary comparison plots table -- file format name based on last part of dirname for input vectors of the difference plots (xxxx.v1.v2.xxxx.png)
note plots
envelope PSF convolved vs unconvolved
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
simple PSF@1.2 (~FWHM) convolved vs unconvolved
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
simple PSF@1.2 (~FWHM) vs envelope PSF convolved
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
simple PSF@1.5 (~envelope) vs envelope PSF convolved
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
  • completeness type plots table -- red=baseline (simulated input), blue=comparison for completeness
note plots
envelope PSF convolved vs unconvolved
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
envelope PSF vs simple PSF@1.2 convolved
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
envelope PSF vs sim20 inputs
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
simple PSF@1.2 vs sim20 inputs
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.
Error: Macro Image(.png,200px) failed
Attachment 'wiki:ppStack_testing_201111: .png' does not exist.

MD04 20 input set:

  • previous version (cleaned..) : input=4.5, auto selected target 5.3 (5.4 largest input PSF)
    • manually set target 4.5, selects target seeing PSF to be 5.3 still.
    • manually set target 4.0, selects target seeing PSF to be 4.8. conv PSF grows to 5.3 (5.4 was largest input PSF, related?)
  • revised version (6/30) in main directory testppstack_deep201111/MDdeeptest
    • mem20_stack_test.fixpsfenvel -- normal PSF envelope version
    • mem20_stack_test.fixsetpsffwhm -- simple PSF model sized to FWHM of inputs

need to revise/update MD04

Case 1b: Simple Model FWHM+X Target PSF

If stack quality (TBD) is highly sensitive to deconvolution, could look at simple PSF model set to the FWHM + X, where X could be the input FWHM stdev or stdev/mean.

PSF.OUTPUT.FWHM         F32    -1.0             # Target size for output PSF (if not auto-sized), -1 as FWHM
PSF.OUTPUT.NFWHMSIG     F32    -1.0             # N FWHM-sig to be added to the FWHMmean for PSF, N<0 and then +FWHMstdev/FWHMmean

Case 2: Improved PSF Evelope as Target PSF

  • tests to check doing what expecting of it
  • improve rejection of extremely deviant inputs
  • improve rejection of odd inputs
  • generate envelope based on most common/likely versus all inputs

Monitor Modifications

Should watch results of changes in nightly science products.

SYS.ERR=0 @3/15

MD nightly stacks --

  • rejection 0-1 inputs similar to before the change
  • convol PSF growth before 4.6->7.1 from 3/11, now 4.9->5.3 from 3/19 and growth similar to that seen when manually setting target PSF (related?)

SAS stacks don't use convolved stacks so shouldn't have affect.

LAP not yet running again (3/19) --

Target PSF

More difficult to monitor as little uses the convolved nightly stacks (if anything?). However, if LAP is running then that will be the source to monitor.

Attachments