
20110303

  psphotStack sample timing:

  * convolve images & stamps: 78 sec
  * standard photometry : 250?
  * petrosians : 6 sec
  * ext fits : 350 sec
  * radial aps : 140 sec

20110302

  * total number of images generated by this process:

 load:
 raw   	  : Nfilter x (image + variance + mask) (20M) 
 cnv   	  : Nfilter x (image + variance + mask) (20M) 
 (8 * 4B) + (4 * 2B)

 match: (added)
 match 	  : Nfilter x (image + variance + mask) (20M)
(4 * 4B) + (2 * 2B)

 phot: (added)
 chisq 	  :       1 x (image + variance + mask) (10M) (freed at end of psphotStackReadout?)
 backgnd  :       1 x (image) (10M) or Nfilter (20M) [unsure]

 load:
     1089174         4000000    psFitsImage.c:467
     1092246         4000000    psFitsImage.c:467
     1095902         4000000    psFitsImage.c:467
     1098974         4000000    psFitsImage.c:467
     1102619         4000000    psFitsImage.c:467
     1105669         4000000    psFitsImage.c:467
     1109303         4000000    psFitsImage.c:467
     1112353         4000000    psFitsImage.c:467
     1090766         2000000    psFitsImage.c:467
     1097494         2000000    psFitsImage.c:467
     1104200         2000000    psFitsImage.c:467
     1110884         2000000    psFitsImage.c:467

  match:
     1128976         4000000    pmSubtractionMatch.c:178
     1128979         4000000    pmSubtractionMatch.c:183
     1652553         4000000    pmSubtractionMatch.c:178
     1652556         4000000    pmSubtractionMatch.c:183
     1128982         2000000    pmSubtractionMatch.c:189
     1652559         2000000    pmSubtractionMatch.c:189

  
     2132413         4000000    pmFPAfileDefine.c:1275
     2133004         4000000    pmFPACopy.c:71
     2133010         4000000    pmFPACopy.c:71
     2133007         2000000    pmFPACopy.c:71
     8614548         2000000    psBinaryOp.c:502
     8617313         2000000    psBinaryOp.c:502

20101221 

  psphotStackReadout is now correctly subtracting the PSF models from
  the images as it measures the radial aperture fluxes. 

  Some issues:

  * the source properties get buggered up by the radial aperture flux
    analysis (we need to re-determine the psf, re-create the source
    pixels, and re-fit the models (linearly) to subtract them
    correctly).

  * the standard analysis sequence is not doing a second pass

  * the choice of the best model is ambiguous

  * the radial aperture (and source addition / subtraction) is only
    using the PSF model

  * make sure psphotEfficiency actually subtracts the inserted fake
    sources

  * XXX how does the PSF-convolved model work with changing psf models?

20101207

  header fields for PSPS:

  * stack needs to count # of inputs
  * propagate the input header to the output image (DONE)
  * save PSF parameters in header 
  * stack type is only known to the launcher

  * what are we doing for the stack calibrations??

skycellID    : code
surveyID     : code
filterID     : code
stackMetaID  : stack_id 
photoCalID   : photcode -> number
magSat       : FSATUR       saturation magnitude level ** not correctly set
completMag   : FLIMIT       95% completion level in mag ** not correctly set 
stackTypeID  : STK_TYPE     stack type identifier ** deep stack, nightly stack, best IQ stack?
refImageID   :   	    identifier of image used as reference for analysis
subtrImageID : N/A (stack)  identifier of image subtracted to generate difference image
analVer      :   	    analysis version index  ** index for tess_id + skycell_id + filter?
nP2Images    : NINPUTS 	    number of P2 images contributing to this cell ** missing from input stack
astroScat    :   	    astrometric scatter for chip ** measure scatter on stack creation?
photoScat    :   	    photometric scatter for chip ** internal scatter?  
nAstroRef    :   	    number of astrometric reference sources ** (connected to above)
nPhoRef	     :   	    number of photometric reference sources ** same
psfFwhm	     :   	    PSF full width at half maximum ** 0.5(FWHM_MAJ + FWHM_MIN)
psfmodelID   : * PSFMODEL   PSF model identifier ** save the PSF model name in the header
psfWidMajor  : * FWHM_MAJ     PSF parameters
psfWidMinor  : * FWHM_MIN     PSF parameters
psfTheta     : * ANGLE        PSF parameters
psfExtra1    : * PSF_EXT1   PSF parameters ** (at field center?)
psfExtra2    : * PSF_EXT2   PSF parameters ** (not set for all models)
photoZero    :   	    local derived photometric zero point 
photoColor   :   	    local derived photometric color term
ctype1	     : * CTYPE1  	    name of astrometric projection in RA ** propagate from input stacks
ctype2	     : * CTYPE2  	    name of astrometric projection in DEC
crval1	     : * CRVAL1  	    RA corresponding to reference pixel
crval2	     : * CRVAL2  	    DEC corresponding to reference pixel
crpix1	     : * CRPIX1  	    reference pixel value for RA
crpix2	     : * CRPIX2  	    reference pixel value for DEC
cdelt1	     : * CDELT1  	    scale factor for RA
cdelt2	     : * CDELT2  	    scale factor for DEC
pc001001     : * PC001001	    elements of rotation/Dcale matrix
pc001002     : * PC001002	    elements of rotation/Dcale matrix
pc002001     : * PC002001	    elements of rotation/Dcale matrix
pc002002     : * PC002002	    elements of rotation/Dcale matrix
polyOrder    : * NPLYTERM	    polynomial order of astrometry fit ** default to 1
pca1x3y0     : * PCA1X3Y0     polynomial coefficients for the astrometric fit
pca1x2y1     : * PCA1X2Y1     polynomial coefficients for the astrometric fit
pca1x1y2     : * PCA1X1Y2     polynomial coefficients for the astrometric fit
pca1x0y3     : * PCA1X0Y3     polynomial coefficients for the astrometric fit
pca1x2y0     : * PCA1X2Y0     polynomial coefficients for the astrometric fit
pca1x1y1     : * PCA1X1Y1     polynomial coefficients for the astrometric fit
pca1x0y2     : * PCA1X0Y2     polynomial coefficients for the astrometric fit
pca2x3y0     : * PCA2X3Y0     polynomial coefficients for the astrometric fit
pca2x2y1     : * PCA2X2Y1     polynomial coefficients for the astrometric fit
pca2x1y2     : * PCA2X1Y2     polynomial coefficients for the astrometric fit
pca2x0y3     : * PCA2X0Y3     polynomial coefficients for the astrometric fit
pca2x2y0     : * PCA2X2Y0     polynomial coefficients for the astrometric fit
pca2x1y1     : * PCA2X1Y1     polynomial coefficients for the astrometric fit
pca2x0y2     : * PCA2X0Y2     polynomial coefficients for the astrometric fit
calibModNum  :   	    calibration modification number
dataRelease  :   	    Data release

20100506:

  we may load RAW and/or CNV images.

  * options->convolve:
    * if only one of RAW or CNV exists, convolve it to match target PSF
    * if both exist, convolve desired one
    
    * if matching PSF exists, use it (unless told to re-measure)
    
    * output goes to OUT (which is then used by psphot analysis routines)

  * detect

    * if (RAW) ? RAW : OUT
    
  *   


20100503:

  psphotStack : outline

  * load N images
  * make chisq detection image
  * detect sources on all (N+1) images
  * convolve N input images (to match, to target)
  * measure source properties on convolved images

20100126:

  * watch out for psphotSetMomentsWindow & MOMENTS_SX_MAX,etc
  * watch out for psphotSignificanceImage.c:,psphotEfficiency using the FWHM_MAJ from psphotChoosePSF

20100120 : more stack processing mods:

  there are a number of data collections generated during psphotReadout:

  * detections (currently a structure containing multiple arrays)
  * sources (a psArray)
  * psf (the psf model structure)
  
  * new sources vs old sources:
    there is a sequence on the second pass in which we need to distinguish the sources
    from the first pass from those on the second pass:

    - add noise (old sources only)
    - find detections
    - subtract noise (old sources only)
    - generate sources (new detections only)
    - source classifications (new sources only)
    - guess models (new sources only)
    - replace sources (old sources only)
    - merge sources (new + old -> sources)

    currently we are distiguishing the old vs new based on different arrays.
    can we use the processing flags to distinguish the these cases and carry around 
    only a single source list?

    * detections->peaks holds the most recently detected set of peaks
      detections->oldPeaks holds the previous collection

      detections->footprints holds the full set of merged footprints,
      including assigned peaks from the old and new set.

20100107 : updates for stack processing

  I am building a new version of psphot that operates on a set of
  images (eg, multi-filter or multi-epoch).  The assumptions are:

  * each input image represents the same sky pixels : they are warped to a common frame.
  * each image has been previously processed, with the background
    subtracted (and the psf model determined?)
  * all planes are / can be loaded into memory at once (otherwise I
    need to add a lot of I/O layers)

  psphot currently expects the pmFPAfile of interest to be available in
  config->files with the name PSPHOT.INPUT.  To update the code for
  stacks, I am extending this concept with the pmFPAfileSelectSingle
  API: config->files may contain multiple PSPHOT.INPUT entries, and
  functions which access this file / these files need to specify
  *which* image they want.  The number of PSPHOT.INPUT entries is
  saved in config->arguments as PSPHOT.INPUT.NUM (this can be
  generated from the filerule PSPHOT.INPUT for extension).

  As a result, we need a number of wrapper functions which loop over
  all PSPHOT.INPUT.NUM entries and perform a particular operation on
  one of the entries.  Here are the functions which I have modified
  in this way (function -> child)

  * psphotAddPhotcode -> psphotAddPhotcodeReadout
    * note that the photcode is now saved on readout->analysis
  * psphotSetMaskAndVariance -> psphotSetMaskAndVarianceReadout
  * psphotModelBackground -> psphotModelBackgroundReadoutFileIndex
  * psphotSubtractBackground -> psphotSubtractBackgroundReadout

  side notes on 
  
  * psphotModelBackground vs psphotBackgroundModel : I've renamed
    psphotBackgroundModel (used only by ppStack for now) to
    psphotModelBackgroundReadoutNoFile.  I've also named the
    single-readout version used by psphotModelBackground to
    psphotModelBackgroundReadoutFileIndex.  These two functions do the
    same thing, but psphotModelBackgroundReadoutNoFile does not save
    a pmFPAfile on config->files


20090606 : design notes on the multi-image photometry analysis

  assumptions:
  
  * each input image represents the same sky pixels : they are warped to a common frame.

  * each image has been previously processed, with the background subtracted and the psf model determined

  basic outline:

  * load images, masks, variance : for now, we should load all images.  in the future, we might be able to split this into segments.  

  * option: model and subtract background for each image (should not be needed)

  * option: smooth each image with a psf or fraction (not needed if stack smooths too much?)

  * generate the significance (chisq) image: X_i = sum(f_i^2 / var_i)

  * perform the peak detection on X_i

  * for each of the input images:

    * generate the significance image S_i = f_i^2 / var_i 
    
    * perform the peak detection on S_i

  * merge the list of peaks

  * generate the footprints

  * linear fit to the peaks with the set of images (psf for each image)


  design issues:
  
  * pmSource represents the analysis for an object on a single image
  * extended pmSourceSet to group connected pmSources on multiple images?
