List of tasks / checks to be done for SAS40

SAS39 verification / list of problems to be corrected

• list of issues from Bill's list ( ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/ippToPspsSAS39BillsNotes )
• list of issues from Chris's list ( ippToPSPSSAS39CZWNotes )
• list of issues from SAS39 pspswg/ dravg pages
• list of issues from confluence

Schema 22 changes

Schema 22 views

• problematic views (PSPS)
• creation of new views / tweaking of new views (HAF/KCC)

Addstar Changes necessary

• forced gal stuff for addstar (EAM)

DVOdb calibration issues ??

• do we have any final issues of dvo calibration (EAM)

Addstar/ippTools/IppToPsps code issues

• need to do forced gal addstar / addtool /addRun
• need to do merge of forced gal
• need to finalize forcedGal schema for 22 and send to conrad
• need to do ipptopsps for forcedGal
• need to make fullforcewarps be less stupid in ipptopsps
• redo addstar for all stages

pv3 / non sas40

• ipptopsps mega batches ?
• crad's fits <-> csv thoughts

schema changes

• MeanObject:xqfPerfect is always -999 asana
  • it is set from PSF_QF_PERF_MAX in cps table in ipptopsps.
  • NOT set in the DVO cps table, though relphot code seems to set it: re-run relphot and fix (EAM)
• FrameMeta: remove p1Recipe, p2Recipe, p3Recipe asana, done
• FrameMeta: add md5sum (from smf)- make it a varchar of reasonable length (10?) asana
• ImageMeta: psfModelID - should be derived from smf header ("3" for PS_MODEL_PS1_V1) asana, done
  • match smf / cmf PSFMODEL to entry in FitModel Table and use the value
• FitModel: fill in correct models. asana, done
  • EAM: I updated table.IN.vot to have all of the real models
  • EAM: I added parameter descriptions, removed table columns for param11-39
• ImageMeta:detrend4-8 fill with NOISEMAP, NONLIN, VIDEODARK (7,8 left blank) asana, done
• (alltables):dataRelease changed to processingVersion and to use 3 for PV3 asana, done
• (alltables):calibModNum - drop. It's in ImageMeta, might be in others. asana, done
• DiffMeta:posImageID, negImageID need more documentation asana, done
• WarpToImage: need to add this table (HAF) asana
• DiffDetection: find -999 columns and list them (WS) asana, done
• DiffDetObject:html,zoneID,cz,cy,cz,lambda,beta,l,b to be filled by PSPS (CH/TC) asana, done (TBR)
• DiffDetObject:description of table is vague and needs to be cleaned up (HAF/EAM) asana, done
• DiffDetObject:xqfPerfect is always -999 asana
  • same as for MeanObject above: fix in relphot (EAM)
  • it is set from PSF_QF_PERF_MAX in cps table in ipptopsps
• StackMeta:magsat - drop it, we do not calculate (in detection/diff/ff as well) asana, done
• StackMeta:completMag - detectionThreshold --> DETEFF.MAGREF (instrumental mag, needs zp / exp correction for stack and detection and diff) asana
• StackMeta: astroScat - AST_CDX, AST_CDY calc= sqrt(a2+b2) - this is not the astrometric error - a measurement of the calibration but not definitive asana
• StackMeta:photoScat - ZPT_ERR - describe it better: the scatter of the calibration stars asana
• StackMeta:nAstro - NASTRO : change to nRefStars asana
• StackMeta:nPhoRef - (delete) asana, done (HAF)
• StackMeta: add set of similar names (same information, from relexp/relstack) recalAstroScat, recalPhotoScat, recalNAstroStars, recalNPhotoStars asana
• StackMeta:calibModNum - remove from schema asana, done
• StackObjectThin:stackDetectRowID (WS/HAF/EAM) is it set correctly asana
• StackObjectThin:primary not set (we understand, from dvo addstar -list bug) asana, done
  • addstar name is fixed in trunk, primary should now work (EAM: check for SAS40 calib & 3pi)
• StackObjectThin:bestDetection not set (???) EAM??? asana
• StackObjectThin:xInfoFlag2 not set, looks like bug in ipptopsps (HAF) asana, done
• (alltables):(allfluxes) convert to jansky - need conversion from EAM asana, done
• ForcedMeanObject:xGamma drop asana, done
• ForcedMeanLensing:xpsfE1 xpsfE2 are -999 asana, done
  • these fields should not exist -- the per-image values psfE1,psfE2 do not have mean equivalents
• ForcedWarpMeta:skycellID to be split into projectionCell and skycellID as done for other tables (HAF) asana
• ForcedWarpMeta:photocalID (???) EAM??? asana
• ForcedWarpMeta:analVer should be set to svn number asana, done
• ForcedWarpMeta:astroScat - drop (HAF) asana, done
• ForcedWarpMeta:photoScat - same as in stack (HAF)
• ForcedWarpMeta:nAstro - drop (HAF) asana, done
• ForcedWarpMeta:nPhoRef - same as in stack (HAF) asana
• ForcedWarpMeta:magSat -drop (HAF) asana, done
• ForcedWarpMeta:completMag -- as done for stack/p2 asana. done
• ForceWarpMeta:add 2 columns recalPhotoScat, recalPhoRef (and grab from relexp (fullforce) info for photoscat/ nphoref) asana
• StackModelFitDev/StackModelFitExp/StackModelFitSer : postions are inconsistent -- all should be RA,DEC (with 8 byte doubles) asana, done

ipptopsps issues from pspswg page

• missing zp in stackobjecthin/stackobjectattributes mags/ fluxes asana, done
• Nigel: primaryflag not set - Gene knows the cause of the problem. asana, done
• Nigel: due to the above, forced mean photometry includes primary and secondary measurements into the averages. asana
• Nigel: in ForcedMeanObject FluxStd is in ADU/sec but Flx and FlxErr are in Jansky. asana, done (all fluxes in Jy)
• Nigel: quick note on naming conventions. Detection and ForcedWarpMeasurement have the same names for columns. Is that OK? (HAF to rename) asana, done
• Nigel: there is a population with halflightrad < 0.1. This is not a star/galaxy issue (using SDSS types, everything with rad<~2 is a star). Colours show density of points. asana
  • HAF/KCC: as listed in Schema 21 the halflight radius is not treated consistently. There is momenRH (labeld as half light radius, sometimes with DROP), in Detection, diffDetection, ForcedWarp, and Stack. This comes from momentRH in the cmf files. We also have a true halflightrad in stack, which is filled from the cmf (HALF_LIGHT_RADIUS. We need to check that this is present in all types of cmf files and use HALF_LIGHT_RADIUS where it was measured and drop momentRH everywhere.
• Nigel: psfMajor/MinorFWHM are in pixels but the schema says arcsec. asana, done
  • HAF: ipptopsps grabs it directly from dvo (no conversions, i see min/majorFWHM is in stack, detection, diff, forcedwarp) - is it wrong for everything or only certain tables? If only certain tables, EAM needs to apply something in dvo. If it's wrong for all, HAF needs to apply correction in ipptopsps. (see other examples, in ipptopsps)
• Roberto: In ForcedMeanObject all g,r,i,r,y-Gamma values are still -999. asana, done
  • EAM: these are not valid and should be dropped
• apflux and apfluxerr still follow multiple, noise-free sequences in all filters. asana, done
• The values of apfluxstd are strange, why can they have the value of -999 when apfluxerr is 'normal'? asana
  • This happens when nTotal>1: for example, obID=ForcedMeanObject in the gband has gnTotal=4, gApFlux=8.19722E-06, gApFluxErr=3.59973E-07 and gApFluxStd=-999.
  • HAF: these come directly from the cmf, no other translations done - EAM? EAM will look at
  • EAM: I investigated these: the values are correctly inherited from the cmf. the trends are correct except that the apFluxErr uses sqrt(image) instead of sqrt(variance).
• Roberto: in StackModelFitDeV the DeVCf are all -999 in all filters (drop). asana, done
• Roberto: in StackModelFitExp ExpCf are all -999. (drop) asana, done
• Roberto: in StackModelFitExp, there are still 'sensible' fits (example: objID=101703338181088339, gband) where RAexp and DECexp are -999 asana
• Roberto: in StackModelFitSer ExpCf are all -999. nuerr=0.1 always asana, done
  • HAF CF is not loaded for Dev/Exp/Ser in ipptopsps, need to investigte if in cmf. EAM : not measured, drop it
• Roberto: in StackModelFitExtra logC and logA are -999 asana, done
  • HAF logC I couldn't find a column for in the CMF, logA is loaded by ipptopsps so must be null in the smf file. EAM: not measured, drop them
• Roberto: in StackPetrosian many petRadiusErr are -999, although all the other errors are not (example: objID=101663337788271850, gband) asana
  • HAF PetRadErr is loaded in ipptopsps, might be null in cmf (EAM?) EAM to investigate
• Roberto: in ForcedWarpMeasurement psfLikelihood is always -999 asana
  • HAF: loaded in ipptopsps: might not be set in CMF (EAM?) EAM: do we have a chi sq? calculated in ipptopsps from a chi sq (HAF to investigate)
• Roberto: in Detection the apflux and apfluxerr still follow multiple, noise-free sequences, errors on kron fluxes don't asana
  • HAF: EAM investigate?
• Roberto: in DiffDetObject cx, cy,cz, l, b, lambda, beta are -999, raErr and DecErr are 0 asana, done
  • HAF: cx/cz/cy/l/b/lambda/beta should be calculated by PSPS.
  • HAF raErr/decErr sounds like dvo calibration problem (bug EAM, ipptopsps tries to load) HAF: need to remove ra/dec err does not make sense for diff
• Roberto: in DiffDetection psfMajorFWHM, psfMinorFWHM, momentM4C, momentM4S, momentR1, momentM3C, momentM3S are -999 asana
  • HAF these are commented out in ipptopsps: I think they may not actually be set, I need to investigate the CMF.

issues on confluence page

• lambda,beta,l,b,gQfPerfect,rQfPerfect,iQfPerfect,zQfPerfect,yQfPerfect are set to -999 in DiffDetObj asana
• psfMajorFWHM, psfMinorFWHM, psfCore are set to -999 in DiffDetections asana
• psfQfPerfect = 0 for all g band detections in DiffDetections, but ~0.99 for the other filters asana
• MomentM* are set to -999 in DiffDetections. MomentM* might be ill defined since it is a diffim with negative weights. asana
• Why is obsTime in DiffDetections slightly different than the obsTime in ForcedWarpMeasurement? It looks like obstime in ForcedWarpMeasurement is the same as in the Detections table. obsTime in DiffDetections is ~0.0006 days = 50 seconds earlier than in ForcedWarpMeasurement and Detections. asana
• in diffMeta, there is a diffMetaID. In DiffDetection, there is a diffDetMetaID. It seems like that these refer to the same attribute, so shouldn't it be named the same? asana
• zp=25 in DiffDetections. Is this correct, i.e. are all diffims normalized to zp=25? asana
• Do we really need moment* and kron* for difference image detections? asana
• Let's pick a random warp. This warp has detections in the warp, forced photometry measurements, and also difference image detections for the diffim using this warp as positive image. It's not clear how to associate these detections. asana

additional schema changes from HAF/KCC IAU booth

• to be added later, on HAF's laptop