= Summary of ppSub_testing_201201 ---IN PROGRESS PLEASE DO NOT EDIT--- = [[PageOutline]] The primary purpose has been expanded and redirected. While it appeared that the fixed convolution direction may have been the likely cause of an overabundance of false detections from residuals of bright sources in the field, issues also in the convolved stack with a resulting larger than selected target PSF has illustrated there may also be issues with the PSF matching/convolution itself. So the scope of testing has been broadened/focused to the PSF matching/convolution in ppSub (may add ppStack examples later) * is ppSub doing what we want * is ppSub doing what we think * has something changed in the data or what the code is doing General tasks (no particular order): * wrap up test and activate the convolution direction choice in ppSub for general use option * test and verify the PSF matching/convolution, include the "no-conv" case as well for close matches * investigate options for the false detections (may be flags2 use, may be solved with PSF matching or convolution direction) * monitor if working as expected and address any additional issues found * establish a baseline to verify against any past/future changes in code and with reprocessing (a test set of real+simulated images) * kernel (other settings) parameter space, == Background == Samples of previous information/work/comparisons * http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ppSub_vs_Hotpants * http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Basic_GPC1_Processing_IPP_Photpipe_Comparison * http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ppSubModifications_201009 * http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ppSub_Modifications_20101021.single * http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ppSub_Modifications_20100918.single * http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/TransientPhotfestSept2010 * http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Performance_Tests_July2010#A.X_Diffim_photometry_reprocessing_data_sample * http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Diffimtest_IPP_photpipe_sne * http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Diffimtest_ut100510_md07s056 = Setup = Operations tag for testing base diffims is ipp-20111220 (since local tests could/should fix at some r since the tag has revisions added to it? ) Branch for originally testing the choice of convolution direction but now full development testing is also updated to r33098 (from 1/11/2012) -- http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/browser/branches/meh_branches/ppsub_test * for convolution direction choice * normalization updated each direction -- need to look if in path for ppStack fixed, seem to recall modifies even fixed (may be for both) * for both * kernel scaling based on Kron radius instead of * general, help for MOPS using DIFF_R_P for cuts * ppSub/src/ppSubFlagNeighbors.c DIFF_R_P/M chooses closest match based on detection peak flux s/n, change to 1/psfmagerr used for DIFF_SN_P/M. * to be tested how responds to if no input catalogs provided -- no catalogs produce an error before the change {{{ Error in subtraction: -> subScaleKernel (ppSubMatchPSFs.c:127): Problem in data values Unable to scale kernel, since no sources were provided. -> ppSubMatchPSFs (ppSubMatchPSFs.c:379): Problem in data values Unable to scale kernel parameters -> ppSubLoop (ppSubLoop.c:113): Problem in data values Unable to match PSFs. }}} = General/Baseline "start" tests = = Kernel Parameter space = HOTPANTS suggested: * need to also resolve size/sigma conversion between ppSub and HOTPANTS M31 group: standard deviations of 9, 6, 3 and 0.1 pixel and a polynomial order of 2, 4, 6 and 0 = PSF Matching/Convolution = == Poor PSF-matching? == Not sure yet what to classify this as other than the PSF not being matched well. Suspect similar to issue with convolved stacks in ppStack. == No-convolution? == Reports from other surveys that diffims are poorer with images having similar PSFs, convolution isn't well matched? Is this for dual only? = Convolution direction choice = We want to avoid large deconvolution in PSF-matching image subtraction. As the image quality of the images continues to improve, always convolving the reference stack will not produce the best difference image and convolving the input image is desired. The quality of the difference image is evaluated in both directions and the better is chosen for the output (need to add specific condition test). MD04 sample: Not all seem to be really due to convolving in the wrong direction and need to identify and separate the cases/causes if can. Running samples forced in both direction useful to see. CfA photpipe/HOTPANTs direction: using single template/reference stack so locally get keyword CONVOL00 from header except slight issue with amps. '''All are convolving the TEMPLATE only''' so also suggest issue seems not a convolution direction but convolution issue itself. * discovered the current photpipe setup is now set to for convolving the template only.. rather then testing both.. argh.. Sample diffims with direction forced and detections marked (note both stack-refstack and stack-stack cases). The stack-stack case is the really the only way to test the convolution direction choice now unless go back to the MD04.V2 sample. (easiest to use event stamps from pages, but may want image photpipe diffim..) * ref - ref * stk - ref vs ref - stk (both directions) * stk - stk (both directions) || '''event (auto direction)''' || '''TARGET2 (convolved ref)''' / '''TARGET1 (convolved stk)''' || '''photpipe automatic (direction)''' || comment || || X (TARGET2) || [[Image(test.png,300px)]] || [[Image(test.png,300px)]] (TEMPLATE) || conv refstk better, || Photometry/detection comparisons: Simulated sample: = Test Sets = Need tests to verify changes will not break diffims for other surveys (MOPS etc) but do not need to worry about magic now. Not for testing if they will choose the right direction (only if using AUTO), but changes involved part of the normalization and will need to verify doesn't change flux/variance values: * MD * simulated image set WW,WS,SS, SSref -- since SSref for MD04 seem to prefer convolving the deeper refstack still, will need to generate * warp-warp diffim: MD and 3PI TTI * warp-stack diffim: MD and LAP stack == MD04 primary samples == Initially selected for if choosing the proper direction in stack-stack diffims now using MD04. Was previously done using MD10, but we want to continue to build up a test set for testing other parts of the IPP and MD04 is a favorite with extensive overlap from other surveys. Start with just 2 skycells having example events from last year for photometry comparisons as well - skycell.055 (previous tessellations 077) -- near center - * http://telescopes.rc.fas.harvard.edu/ps1/alerts2011/2010-C-020101/2010-C-020101.html - AGN@0.345 (xy?) * http://telescopes.rc.fas.harvard.edu/ps1/alerts2011/2010-D-030288/2010-D-030288.html - poor sub/var? * http://telescopes.rc.fas.harvard.edu/ps1/alerts2011/2011-A-120103/2011-A-120103.html - alert * http://telescopes.rc.fas.harvard.edu/ps1/alerts2011/2011-A-120706/2011-A-120706.html - alert and spectrum * new events * http://telescopes.rc.fas.harvard.edu/ps1/alerts2012/2012-A-350113/2012-A-350113.html - residual peak? * http://telescopes.rc.fas.harvard.edu/ps1/alerts2012/2012-A-350492/2012-A-350492.html - bright star * http://telescopes.rc.fas.harvard.edu/ps1/alerts2012/2012-A-350211/2012-A-350211.html - alert skycell.058 (previous tessellations 073) -- edge - * http://telescopes.rc.fas.harvard.edu/ps1/alerts2011/2010-A-000541/2010-A-000541.html - QSO@2.051 * http://telescopes.rc.fas.harvard.edu/ps1/alerts2011/2011-B-130093/2011-B-130093.html - A star * http://telescopes.rc.fas.harvard.edu/ps1/alerts2011/2011-C-140196/2011-C-140196.html - sdss target QSO * new events * QUB cross-match page: http://star.pst.qub.ac.uk/ps1/psdb/crossmatchipptocfa/ (add when time.. difficult to search the new crossmatch page..) Difficult to use past examples of wrong convolution direction since refstacks remade, so select sample to just test current choices. The monitoring part should take care of of this in more recent cases as MD04 actively being observed for next couple months. QUB examples of wrong convolution choice: * http://star.pst.qub.ac.uk/ps1/psdb/userlist/10/ * http://star.pst.qub.ac.uk/ps1/psdb/userlist/5/?sort=-followup_flag_date * http://star.pst.qub.ac.uk/ps1/psdb/candidate/1095456431022404400/ Sample nightly stacks for both skycells with same obsdate to look at variation of PSF center and edge (0.25 "/pixel), limited to r-band for now and a sample size that can be manually rerun with some ease. || '''obsdate''' || '''stack_id''' || '''FWHM_MAJ (pixel)''' || '''IQ_FW1 (pixel)''' || || || || || || || || skycell.055 || || || || refstack.20111012 || 450933 || 5.206454 (+-0.6-0.8) not the 4.5 on the MD.GR0 page.. || 4.08811 (+-0.3-0.4)|| || 20100110 || 444020 || 4.878624 || 3.891729 || || 20100121 || 444079 || 5.843588 || 4.574009 || || 20110129 || 444251 || 4.424473 || 3.378561 || || 20110201 || 644876 || 5.746622 || 4.748873 || || 20110312 || 446658 || 4.958342 || 3.74775 || || 20110315 || 645114 || 4.270276 || 3.232871 || || 20110423 || 446715 || 4.603964 || 3.510957 || || 20110429 || 645474 || 7.178107 || 6.027955 || || 20120115 || 622752 || 5.805826 || 4.622056 || || 20120121 || 646801 || 6.346761 || 5.359003 || || (swap 20120126 for 20120115?) || 676389 || 4.443175 || 3.475093 || || || || || || || || skycell.058 || || || || refstack.20111012 || 450936 || 4.616929 || 3.926292 || || 20100110 || 444023 || 4.777016 || 3.687181 || || 20100121 || 444082 || 5.567342 || 4.243489 || || 20110129 || 444254 || 3.976701 || 3.100284 || || 20110201 || 644879 || 5.023516 || 4.421264 || || 20110312 || 446661 || 5.114515 || 3.701454 || || 20110315 || 645117 || 4.58284 || 3.376984 || || 20110423 || 446718 || 4.358292 || 3.138227 || || 20110429 || 645477 || 6.088856 || 5.447231 || || 20120115 || 622755 || 5.300241 || 4.337034 || || 20120121 || 646804 || 5.627441 || 5.058287 || || (swap 20120126 for 20120115?) || || || || || || || || || == Simulated Samples == Simulated image sample: || '''filename''' || '''sim FWHM''' || '''FWHM_MAJ (pixel)''' || '''IQ_FW1 (pixel)''' || || || || || || || || || || || = Monitoring = == Nightly Tests of Diffims == Is there any conflict with the expected desired direction? Compare to CfA photpipe, detection counts. * having IQ/FWHM keywords in stack images or at least the gpc1 DB would be helpful here since are calculated for the PSF matching in the diffim process, doing manually here takes a good amount of time. Is the convolution not optimal? == HOTPANT Diffim Comparisons == Difficult to automate and may want to force both directions for comparison as well as shallower reference stacks to use. Are the diffims continuing to produce similar results. * for select night samples should setup and run locally. still on the todo list. == diffim stats and flags == There are columns of diffim stats and flags to help characterize the diffim and detections * detection catalog uses CMF_PS1_DV2 -- DIFF_* entries http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/CMF_PS1_DV2 * descriptions need more added * FLAGS2 http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/browser/trunk/psModules/src/objects/pmSourceMasks.h * PM_SOURCE_MODE2_MATCHED for psphotStack * descriptions need more added DIFF_ equivalent setting from photpipe: * DC_FRATIO_MIN 0.75 * DC_NRATIO_BAD_MIN 0.7 * DC_NRATIO_MASK_MIN 0.6 * DC_NRATIO_ALL_MIN 0.5 * DC_NGOOD_MIN 3 DIFF_ initial settings from Maria Teresa's tests: * DIFF_NPOS >5 nPos (n pix > 3 sigma) * DIFF_FRATIO >0.7 fPos / (fPos + fNeg) * DIFF_NRATIO_BAD >0.7 nPos / (nPos + nNeg) * DIFF_NRATIO_MASK >0.7 nPos / (nPos + nMask) * DIFF_NRATIO_ALL >0.5 nPos / (nGood + nMask + nBad)