SA 10 Problems

This is the list of problems encountered with SA10. We also have unresolved issues left over from Sa9Problems, most of which still will remain unresolved until the new schema, but we want to continue to track them.

problems (to be fixed for sa11)

Priority	Problem	Fixer	Fixed?	Extra Notes
	DVO average object flags needed to define object quality	gene	done
	varflag needs to mirror information from average object stack flag bits (good, bad, suspect)	bill/heather	done	mirror ID_OBJ_EXT, ID_OBJ_GOOD, ID_OBJ_GOOD_STACK, ID_OBJ_SUSPECT_STACK, ID_OBJ_BAD_STACK, shifted to right as appropriate
	add cx cy cz back to object table	conrad	yes; The code to populate these have been implemented back in but I need to test with Object batches that contain these columns.	These columns are actually need to be provided by IPP2PSPS and defaulted to zero and populated during merge. Tables columns in IPP2PSPS always need to match PSPS in a one to one relationship.
	add a row for a random variable 0 < random < 1 to the object table	heather/thomas	done
	Object.xStackDetectID values are garbage	bills	fixed in objectbatch.py in trunk	problem still exists in sas 11
	Object stack columns are uninitialized for filters that have no stack detection	gene	requires changes to relphot done
	Fix stack airmass - 0.0	gene	done (addrun.pl reads exposure airmass for stacks)
	Object.xFlags ID_SECF_OBJ_EXT may be wrong	gene	done (no longer including stack det in calculation)
	ipptopsps doesn't handle a reduced number of apertures	heather	done
	The DetectionFlags and StackDetectionFlags table up to date? This can be added later as it's a really small table.	?

future fixes

priority	Problem
after LAP.20120706	airmass slopes not set in ppStack.config / airmass not applied by staticsky calibration
primaryF in stackDetection not set

list of issues with SA10 from PSPSWG with answers

• A: Object tests
  • A1: entries set to weird values
    • 1. projectionCellID: is -999 or 1226 (1226 when ndetections = -999)
      • a. It should be -999. When ndetections = -999 we get into funny stuff, specifically, those with ndetections = -999 are filled from skinnyObjects. Those are created from stack/detection batches. Why aren't they filled? There are a handful that are not filled, likely due to the dvo saying "oh, we don't want this as an object". The majority come from a failed batch on the psps (objid out of range)
        • i. Heather thinks we ought not to do skinny batches (?)
        • ii. if it is -999 anyways is this a column we should keep?
        • iii. WES: The idea of a projection cell (and skycell_id for stacks) has meaning, however for skycell_id at least there are issues for changing the current bogus implementation that no one has had time to look at.
    • 2. varFlag = 0
      • a. This is taken care of for future releases. It will be called qualityFlag, and is a subset of flags from infoFlag to help easily find good or bad objects
    • 3. surveyId = 0
      • a. It should be 0 - there is a definition of surveys, and 3pi is 0.
    • 4. stackVer = -999
      • a. This makes no sense in the current schema and Heather thinks we should remove it
    • 5. nw = -999
      • a. We currently do not have w in the dvo or in ipptopsps
    • 6. consistencyFlag = -999
      • a. This has been brought up and ignored before (see SA9 etc)
    • 7. updated = -999
      • a. We don't use this flag (so then why is it there?)
  • A2: 12% have ra_err > 0.1 deg (same for dec_err), and 70% have ra_err = 0 (same for dec_err)
    • a. For the 12% that have raerr > 0.1, the majority come from (ng+nr+ni+nz+ny) = 0 or only a few suspect the calculation is goofy because there are few points to calculate raerr. (GENE??) gene notes that ra/dec err are in units of arcsecs, and he notices that pretty much everything is within 1 arcsec.
    • a. For the 70% that have ra = 0, I did a count(*) and grouped by objInfoFlag. I repeated this for raerr!=0. Here's what I find: if 0x00090000 (avg position used and fitted) is not set, it has raErr = 0 (this is the majority of things). If 0x0040000 (mean astro could not be measured) is set, may fall into either, with a preference for raerr!=0. why do so many not have 0x00090000? are those entirely the stacks? the majority of those have a detection in a filter (for the raErr=0, and flag = 4194304). If I do a similar query but without the restriction on flags (same raerr = 0), I find a whopping majority have NO ng/nr/ni/nz/ny, ie, they are stacks.
      • We need to make it very clear that raErr / decErr is calculated from an average of single frame detections, and may not be present at all when it is a stack with no or few single frame detetctions
      • EAM update : if we only have stack measurements, and they are forced to match a single input (ie, y-band), all ra/dec values will be identical (because the skycells have the same astrometry and the matched detections use the coordinate of the referring source), thus the stdev will be 0.0.
  • A3: 0.1% have ndetections = -999 (one specific patch in ra & dec)
    • a. This is due to a failed batch (outside of PSPS's ra/dec range) I thought it had been fixed (conrad fixed it for the future)
  • A4: ndetections still not equal the sum of ng+nr+ni+nz+ny : far from it actually. ndetections >= 2xsum (and =2xsum for 97%)
    • a. This is a bug in the code, Heather knows where and how to fix
  • A5: sgsep: ranges from 0 to 181.28 [according to Nigel sgsep in object table is not correct, while in stackdetection it looks more normal)
    • a. EAM : STARGAL_SEP is the max of the extNsigma values from the exposures (P2s)
  • A6: ?stackdetectid: what are they? Some are negative, some have single values (equal the the filterID), and some very positive.
    • a. This has been fixed by Bill for future SAS (still broken in SAS 11)
  • A7: General magnitudes comments: (GENE?)
    • 1. gMeanPSFMag - gStackPSFMag is bimodal: two peaks one at -0.03 and one at 0.12 (clearly separated) and not really the case for other bands
      • a. gene needs to address - likely not for SAS or ingest of LAP this time
    • 2. ?MeanKronMag & ?MeanKronMagerr: they span a ridiculous range (which is not the case for ?MeanPSFMag and ?MeanPSFMagerr, hence the reason to flag it)
      • a. EAM : in DVO, the mean kron mag range is not strange
      • b. EAM : in DVO, the mean kron mag error is apparently not set (all values NAN) -- not calculated?
      • c. HAF: meankronmag is set properly in SA11. there are a handful (order of 100) that have ridiculous range.'
    • 3. ?StackPSFmagerr: covers an even more ridiculous range (- 1E59 to +1E59)
      • a. EAM : in the DVO, I find a small fraction of objects with (eg) gStackPSFfluxerr / gStackPSFflux > 0.5 (48 / 200k in my test). of these, a handful have values > 5.0. looking in detail, I've found 2 cases (so far):
        • i. forced photometry on the stack, and the error / flux is reasonable, like 0.6 (not a problem)
        • ii. an object with a bright g-band stack measurement and a second coincident, much fainter measurement. this is a measurement of the wing of the star after it was subtracted. The problem is that the object stack photometry uses the faint, garbage value and not the bright value.
      • b. HAF : stackpsfmagerr is 99% properly set in SA11. There are a handful (order of ~3000) that have ridiculous range, compared to the >1 million or so in SA10
• B. Test on SAS10 stackdetection table with 1% of the data
  • B1. entries set to weird values
    • 1. stackTypeId is always 1
      • a. suspect this is the stack type from IPP
      • WES: All of SAS is deep stacks, not nightly or high image quality. Hence they should all have the same value. Don't know where this is defined.
    • 2. surveyID is always 0
      • a. it should be because it is a 3pi like survey
    • 3. primaryF is always 0
      • a. we don't set it (should we? )
    • 4. x/yposerr: -999 to 17k while x/ypos 10 to 6.3k
      • a. EAM : x & y position error are actually based on the fwhmMajor and fwhmMinor values and the S/N: x = fwhmMajor / (S/N * 2.35), y = fwhmMinor / (S/N * 2.35). these are reasonable (but note the angle compared to the x,y axis), but go to interesting values for very low S/N objects or those with negative fluxes. perhaps it should saturate at 0.5*ap_radius_raw?
    • 5. raerr & decerr: quantized in 0.25 steps + in silly units (not the same as xposerr)+ negative from time to time (not just -999) + range is huge (-8000 to 8000)
      • a. gene to investigate - note that ra/dec err are in units of arcsecond
    • 6. apFluxerr: -999 (i.e. never set)
      • a. noted before on previous SASs. not to be fixed yet
      • b. EAM : relatedly, apFlux is apparently coming from the apMag, not the measured apFlux, so it is NaN for <= 0.0 entries
      • c. EAM : both of these issues were fixed in the smf/cmf output format PS1_SV2, but unfortunately the processing to date is using PS1_SV1 -- we will stick with PS1_SV1 for this round
    • 7. kronrad: min value ~5 : is this expected?
      • a. WES: psphot calculates a minimum value for the kron radius as some function of measured image quality. So a minimum value is expected.
    • 8. psfchisq: 1k to 5k typically.... peak at 2.8k: what's the meaning of that number
      • a. EAM : psfchisq is reporting the chisq, not the chisq per degree of freedom. it should be changed to the latter by dividing by the field PSF_NDOF in the cmf
    • 9. activeFlag: always 0
      • a. we already know about it, if we don't use it can we drop it
    • 10. assocDate: what does it correspond to
      • a. Heather -seems to correspond (at least the day part) to when it got stuffed into psps. Conrad would know more...
    • 11. historyModNum: always 0
      • a. not used
    • 12. datarelease: 10 (ok I presume, as it corresponds to SAS10)
      • a. yes
  • B2. ProjectionCellID do not match between stackdetection and object: not a single one match! Which one is the correct one?
    • a. stackdetection is the correct one - we have the column in Object but we do not use it
  • B3. sgsep do not match between stackdetection and object [see above]; Nigel has the feeling sgsep in stackdetection behaves more as expected.
    • a. The object one comes from the dvo averages table. May need Gene to explain
  • B4. projectionCellID == skycellID: is this normal or is this an artefact of SAS10 being small?
    • a. This is a known issue and not to be resolved yet (why?) WES: A proposal was made to fix the stackMeta.skycellID but there are PSPS validation issues that must be worked out and no one has had the time to investigate. I think it involves creating an init batch with every possible value of skycell id.
  • B5. psflikelihood: why <0 for 10% of the stackdetections and -999 for ~25% of the stackdetections
    • a. ? Gene???

• C. PSPS flags:
  • C1. objectxflags: what does ID_SECF_HAS_STACK correspond to: forced photometry on the stack if set? Not clear why it should indicate whether there is a stack photometry as if there isn't any, then the corresponding photometry column should be set to -999
    • a. ? Gene???
  • D. PSI/PSPS suggestions:
    • D1. It would be good if a small command could be defined for this expression int_ran = (ippobjid % 10) *1000 + ((objid / 10000) % 10)*100 + ((ippobjid / 10) % 10)*10 +(objid % 10) . The neat part of int_ran is that it runs from 0 to 9999 and is randomly distributed. It can be selected from nearly all tables directly in a repeatable fashion. int_ran is very quick to compute (barely increases the runtime of the query). Equally important (if not key), one can match tables which were selected on the same int_ran condition, making it far superior than any true random number associated with any table. The only flaw is that one cannot select any fraction smaller than 1/10000, i.e. 0.01%, but I believe this will rarely be of interest.
      • a. Conrad?
    • D2. Using the query builder on IPP to select all entries in the object table result in a crazy column layout: the first few columns end up being o_objID, o_b, o_ni, o_yMeanPSFMagErr, o_gFlags,.... which is clearly unpractical. This doesn't happen with PSVO, which lists all columns according to the schema browser layout (i.e. standard one).
      • a. Conrad?
• More Issues:
  • 1. Matching stackdetection and object: for a few objects matched on objid, they don't have the same ippobjid (<0.1%) [found this even on a 10% random sampling, because the random number uses ippobjid and objid]
    • a. needs investigation.. Heather/Gene???
  • 2. select * from Detectionflags order by value: query does not return the same flags as listed on PSPSflags_sas10
    • a. this is on heather's list to be fixed
  • 3. objectflags and stackdetectionflags tables are inexistent, i.e. one can not retrieve from the database the meaning of the flags. This should be contrasted with: select * from detectionflags for the detection table
    • a. this is on heather's list to be fixed
  • 4. Object table flags:

        -1. If stackphotometry exist (xFlags & 0x20 !=0), why does xStackKronMag or xStackPSFMag == -999? 
           We have for g r & i:
           0.5%, 8.5% in g (PSF, Kron)
           0.7%, 5.5% in r (PSF, Kron)
           0.7%, 4.5% in i (PSF, Kron)
       - 2. What is the difference between:
           - ID_OBJ_EXT (in object.infoflag) and ID_PSPS_OBJ_EXT (in object.xflags) ? 
           - ID_SECF_USE_SYNTH  and ID_SECF_USE_UBERCAL in object.xflags?
       - 3. Object table flags, either in object.objinfoflags or in object.xflags, do not list flags for the following information:
           - whether the object is detected on the stack or on individual frame. 
             The flag xflags.ID_SECF_HAS_STACK only tells whether the object has a stack photometry or not 
             and as mentionned above a fair fraction of objects with the flag set do not have proper stack photometry (-999).
           - whether the stack photometry for the object is forced or not per band
           - whether quantites come from stack or individual frame (e.g. ra & dec)
           - wheteher photometry was attempted or not (assuming Kron and PSF are automatically done at the same time; 
             otherwise one need to have a flag telling whether psfphotometry / kronphotometry was attempted or not) 
       - 4. Object table need to list the number of detections vs the number of times a position was looked at (for a given survey). This is an issue for non-detection / false positives. Additionally the number of detections should not just be the sum of all detections scrambled together, but either the number of detection or the number of stack detections. In an ideal world, different columns for ndetections and nstackdetections should be listed.
       - 5. The provenance of many flags in the object table is not clear: 
           - where does xflags.ID_PSPS_OBJ_EXT come from?
           - where does xflags.ID_OBJ_GOOD come from? 
       -  6. Unclear whether the following flags are of much use in object.xflags (poor documentation might be a reason to fall in this category):
           ID_SECF_STAR_FEW    = 0x00000001, // used within relphot: skip star
           ID_SECF_STAR_POOR   = 0x00000002, // used within relphot: skip star
           ID_SECF_USE_SYNTH   = 0x00000004, // synthetic photometry used in average measurement
           ID_SECF_USE_UBERCAL = 0x00000008, // synthetic photometry used in average measurement
           ID_SECF_HAS_PS1     = 0x00000010, // PS1 photometry used in average measurement
           ID_PHOTOM_PASS_0    = 0x00000100, // average magnitude calculated in 0th pass
           ID_PHOTOM_PASS_1    = 0x00000200, // average magnitude calculated in 1th pass
           ID_PHOTOM_PASS_2    = 0x00000400, // average magnitude calculated in 2th pass
           ID_PHOTOM_PASS_3    = 0x00000800, // average magnitude calculated in 3th pass
           ID_PHOTOM_PASS_4    = 0x00001000, // average magnitude calculated in 4th pass
           ID_SECF_OBJ_EXT     = 0x01000000, // DVO location of "extended in this band" For PSPS this bit is moved
       - 7. Unclear whether the following flags are of much use in object.objinfoflags 
      (poor documentation might be a reason to fall in this category; note that I have remove all those not any longer set):
           ID_STAR_FEW       = 0x00000001, // used within relphot: skip star
           ID_STAR_POOR      = 0x00000002, // used within relphot: skip star
           ID_OBJ_EXT_ALT    = 0x02000000, // extended in external data (eg, 2MASS)
           ID_OBJ_GOOD_ALT   = 0x08000000, // good-quality measurement in  external data (eg, 2MASS)