== This page discusses the processing stages and database tables to be used to manage IPP full force photometry == Notes on the new "full force" stages of the pipeline The current terminis of staticsky processing is the skycal results. The granulatity of these results are one per filter per skycell per processing version. The new full force stages of the pipeline will take certain static sky parameters back to the input exposures and measure psf and extended source photometry. The input parameters for the full force photometry are contained in a 'cff' file. For efficiency these will be generated during skycal processing. An optional processing stage is outlined here which may be used to generate cff files post skycal (in case algorithms change of tweaks to the processing are necessary). After full force has been run for all of the warps for a given skycell and filter, the results are then summarized though a TBD process. === Tables === Here are my ideas for the structure of the tables. {{{ CREATE TABLE fullForceRun ( ff_id BIGINT NOT NULL AUTO_INCREMENT, warp_id BIGINT, skycal_id BIGINT, sources_path_base VARCHAR(255), state VARCHAR(64), workdir VARCHAR(255), label VARCHAR(64), data_group VARCHAR(64), dist_group VARCHAR(64), reduction VARCHAR(64), registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- time run was registered PRIMARY KEY(ff_id), KEY(state), KEY(label), KEY(data_group), KEY(warp_id), KEY(skycal_id), FOREIGN KEY(warp_id) REFERENCES warpRun(warp_id), FOREIGN KEY(skycal_id) REFERENCES skycalRun(skycal_id) ) ENGINE=innodb DEFAULT CHARSET=latin1; CREATE TABLE fullForceResult ( ff_id BIGINT, path_base VARCHAR(255) NOT NULL, dtime_script FLOAT, quality SMALLINT NOT NULL, hostname VARCHAR(64) NOT NULL, software_ver VARCHAR(64), fault SMALLINT NOT NULL, PRIMARY KEY(ff_id), KEY(fault), KEY(quality), FOREIGN KEY(ff_id) REFERENCES fullForceRun(ff_id) ) ENGINE=innodb DEFAULT CHARSET=latin1; }}} This structure has to a separate run for each skycell. This will lead to a Nstack (~2Million) * Ninputs (~12) runs for a processing version (PVn) Another option would be to structure like diffRun with a input table. In that case skycal_id, and input_path_base would move there. At this level we're really doing the processing on skycells, so the grouping by exposure might be less interesting. Avoiding the extra table simplifies code. Note that this structure ties the full force run to a warp. One could imagine doing running it on a chip, diff, or stack. I'm hesitant to use the stage, stage_id style structure (addRun, distribution) because that makes everything more complicated. The code sharing ends up being somewhat illusory. Are five 10 line functions really better than one 50 line function with lots of cases? ---- We might as well anticipate that we will need to generate cffs as a post skycal step. We don't want to require this because if we can generate it at the skycal stage we will save an awful lot of I/O. {{{ cffGenerateRun cffg_id skycal_id label reduction etc }}} When processing this run the skycal cmf and generates a cff file. Why bother. Why not just create a new skycalRun? Because the skycalRun might have already been used to build psps, etc and so we probably shouldn't confuse things by reruning calibration. ---- Once all of the fullforce warp processing is done we need to somehow collect the results {{{ ffullForceSummaryRun ffs_id skycal_id (sets stack_id, filter, skycell_id) The inputs are defined in the secondary table fullForceResultsInput ffw_id (all inputs are linked back to the same skycal_id insured by query) }}} When this thing is processed the results from the multiple warps are analyzed and we magically get the parameters for any object in the visible universe ;) (Kind of like addstar and dvo). ---- == TOOL == As usual we will define a tool for managing the tables. {{{ fftool - full force tool modes -definebyquery define fullforce runs usual args to select warps args to select input from skycal run or cffgenerate run -select_skycal_label -select_skycal_id -select_cffgenerate_label if neither of these are specified assume -select cffg_id input is in the skycal results -set_ arguments for all other parameters -addresult -revert -updateffrun -definecffgenerate define a cffGenerateRun which will make a cff file for a stack from the skycal outputs -select_skycal_label -select_skycal_id -set_ arguments for all parameters -addcffgresult }}} == Processing == {{{ psphotStack output cmf is run through psastro to calibrate zero point and to produce the 'cff' file - the list of objects which will be measured by psphotFullForce The cff is produced by pmSourcesWrite_CFF which is called if a output file "PSPHOT.OUTPUT.CFF" is defined. This file contains a single fits binary table with 1 row for each source. The values in the row depend on whether or not the source is selected for galaxy fitting. The boolean fitGalayx is set to be true if 1. The extended model fits were performed on the source and one of the models was set to be the source's modelEXT and 2. The model params SXX, SYY, SXY, and model magnitude are finite Here are the values for each row in the cff file ID source->seq (IPP_IDET) (XXX: need to make sure that these are set to the same values as the psf extension. Insure that we write cmf first.) X fitGalaxy ? modelEXT->params[PM_PAR_XPOS] : modelPSF->params[PM_PAR_XPOS] Y fitGalaxy ? modelEXT->params[PM_PAR_YPOS] : modelPSF->params[PM_PAR_YPOS] FLUX (fitGalaxy ? 10**-0.4*modelEXT->mag : source->psfFlux) / CELL.EXPOSURE AP_RADIUS source->apRadius KRON_RADIUS source->moments->Mrf*2.5 PETRO_RADIUS source->extpars ? source->extpars->petrosianRadius : NAN FIT_GALAXY see above PSF_STAR source->mode & PM_SOURCE_MODE_PSFSTAR ? true : false R_MAJOR fitGalaxy ? axes.major : 0 R_MINOR fitGalaxy ? axes.major : 0 THETA fitGalaxy ? axes.theta : 0 MODEL_TYPE fitGalaxy ? modelEXT->type : 0 INDEX fitGalaxy && modelEXT->type == sersic : 0 Notes: The flux is converted to counts per second. It would be feasible to fit multiple models for each source but this would require that psphotFullForce and the function that reads cff files to be enhanced to deal with multiple models. Full Force Pipeline ------------------- The full force pipeline consists of two stages. First psphotFullForce is run on all warp skycells for each stack. Then the program psphotFullForceSummary combines the results to get a single set of parameters for each source. psphotFullForce --------------- INPUTS ------ The cff file defines the set of sources to be processed. The data is read by a function in psModules and a source is created for each row. A psf model of the type given by the PSFMODEL keyword is created for each source. X and Y are used to set the parameters. A peak is allocated at the position with detValue, rawFlux, and smoothFlux set to the value of the FLUX column. The flux value (in counts per second) is later multipled by the warp exposure time by the function psphotLoadExtSources. moments are allocated with Mx = X, My = Y, Mrf = 0.4 *KRON_RADIUS tmpFlag PM_SOURCE_TMPF_MOMENTS_MEASURED is set. Note that this leaves the higher orde moments NAN if FIT_GALAXY is set and the model type is a recognized galaxy model type and extended source model is created. The model parameters are set based on X, Y, R_MAJOR, R_MINOR, and THETA. If the model type has an 8th parameter it is set to 0.5/SERISC_INDEX (BUG: this will result in INF if sersic index is zero) The warp image, mask, and variance images are read. If passed as an argument the warp stage psf file is read as well. Currently the script does not pass the psf file because Gene told me not to. I don't recall the reason. Processing ---------- Here are the important steps in full force processing. psphotSetMaskAndVariance (config, view, filerule); // generate a background model (median, smoothed image) if (!psphotModelBackground (config, view, filerule)) { return psphotReadoutCleanup (config, view, filerule); } if (!psphotSubtractBackground (config, view, filerule)) { return psphotReadoutCleanup (config, view, filerule); } if (!strcasecmp (breakPt, "BACKMDL")) { return psphotReadoutCleanup (config, view, filerule); } if (!psphotLoadPSF (config, view, filerule)) { // this only happens if we had a programming error in psphotLoadPSF psError (PSPHOT_ERR_UNKNOWN, false, "error loading psf model"); return psphotReadoutCleanup (config, view, filerule); } NOTE: This is where the flux is multiplied by the exposure time to convert from counts per second to counts // include externally-supplied sources (loaded onto detections->newSources) psphotLoadExtSources (config, view, filerule); // merge the newly selected sources into the existing list (detections->allSources) // NOTE: merge OLD and NEW NOTE: (we only have NEW) psphotMergeSources (config, view, filerule); // construct sources and measure moments and other basic stats (saved on detections->allSources) // all sources use the auto-scaled window appropriate to a PSF, except for the saturated // stars : these use a larger window (3x the basic window) From comment in psphotFullForceSourceStats // construct pixels and measure moments for sources. // NOTE: this function differs from psphotSourceStats in the following ways: // 1) detections->allSources are used instead of newSources // 2) the sources are used directly (peaks are not assigned to sources here) // NOTE: this function is meant to be used by psphotFullForce, and the mode of the loaded // sources should have (MODE_EXTERNAL) set. If so, the Mx,My values will not be recalculated if (!psphotFullForceSourceStats (config, view, filerule, true)) { // pass 1 psError(PSPHOT_ERR_UNKNOWN, false, "failure to generate sources"); return psphotReadoutCleanup (config, view, filerule); } // generate a psf model for any readouts which need one // psphotFullForcePSF (config, view, filerule); // XXX I'm not sure we need a different algorithm here or not : we are supplying the // sources; we should mark with a flag bit the ones we actually want to use as PSF // stars (this means we need to supply this info in the load). psphotChoosePSF (config, view, filerule, false); // Construct an initial model for each object, set the radius to fitRadius, set circular // fit mask. NOTE: only applied to sources without guess models // XXX this currently only generates a PSF model psphotGuessModels (config, view, filerule); // linear PSF fit to source peaks, subtract the models from the image (in PSF mask) // XXX this will fit non-PSF sources if they are supplied psphotFitSourcesLinear (config, view, filerule, false, false); // measure kron fluxes psphotKronFlux (config, view, filerule); // measure petro fluxes psphotPetroFlux (config, view, filerule); // measure radial apertures psphotRadialApertures (config, view, filerule, 0); // measure galaxy shapes psphotGalaxyShape (config, view, filerule); // calculate source magnitudes (psf mag and ap mag) psphotMagnitudes(config, view, filerule); // replace background in residual image psphotSkyReplace (config, view, filerule); Galaxy Fitting -------------- Most of the functions performed by psphotFullForce are the same as standard psphot. psphotGalaxyShape is the function that creates extended models for the objects selected for galaxy fitting over a range of galaxy parameters and measures the chisq, flux, flux error for each trial. The results of this are saved in a new cmf extension table refered to as 'XGAL'. Currently the only parameters varied are the major an minor radii. The range of values is determined by recipe. Currently 49 combinations are tried with the range in axes of 0.85 to 1.15 times the input values in steps of 0.05 * the input values. In processing the source's modelEXT is left equal to the model for the nominal (input values). We do this so that the nominal values are available in psphotFullForceSummary but this yields a less useful residual image. XXX: Perhaps psphotFullForceSummary should just read the CFF file (not as sources but as a table) and get the nominal values from there. OUTPUTS The output of the psphotFullForce consists of a standard cmf file and optionally a residual image. psphotFullForceSummary ---------------------- Once psphotFullForce has been run on all of the warps for a given skycell filter combination the program psphotFullForceSummary is run to combine the results. Inputs ------ The cmfs from the full force runs on the individual warps. Computations ------------ psf measurements petrosian radius and magnitude TBD mean radius? galaxy fits For each R_MAJOR, R_MINOR combination (i) the chisq of the combintation is measured by summing over the measurements on the indvidual warps (k). Note that sources which are masked in an input are not included in these sums. This is determined by requiring that PSF_QF_PERFECT > 0.9 chisq[i] = sum(chisq[k] * nPix[k]) / sum(nPix[k]) flux[i] = sum( (flux[k] / dFlux[k])**2) / sum ( 1 / dFlux[k]**2) dFlux[i] = 1 / sum ( 1 / dFlux[k]**2 ) Once these values are computed the trial with the lowest chisq value is chosen as the "best fit" and the extended source model paramters are set Outputs ------- These values are written to a summary cmf in the standard format }}}