IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 39565 for trunk


Ignore:
Timestamp:
May 1, 2016, 12:20:15 PM (10 years ago)
Author:
eugene
Message:

fix description labels for various info flags (errors noted by Nigel M.); do NOT delete detections outside of cpt file boundaries; do NOT shift SECF_OBJ_EXT bit (0x0100000 to 0x2000)

Location:
trunk/ippToPsps
Files:
2 edited

Legend:

Unmodified
Added
Removed

  • trunk/ippToPsps/config/tables.IN.vot

    r39392 r39565  
    659659          <TR><TD>DIFF_WITH_SINGLE</TD>     <TD>0x00000001</TD><TD>1         </TD><TD>Difference source matched to a single positive detection.</TD></TR>
    660660          <TR><TD>DIFF_WITH_DOUBLE</TD>     <TD>0x00000002</TD><TD>2         </TD><TD>Difference source matched to positive detections in both images.</TD></TR>
    661           <TR><TD>MATCHED</TD>              <TD>0x00000004</TD><TD>4         </TD><TD>Difference source matched to positive detections in both images.</TD></TR>
     661          <TR><TD>MATCHED</TD>              <TD>0x00000004</TD><TD>4         </TD><TD>source generated based on another image (forced photometry at source location).</TD></TR>
    662662          <TR><TD>ON_SPIKE</TD>             <TD>0x00000008</TD><TD>8         </TD><TD>More than 25% of (PSF-weighted) pixels land on diffraction spike.</TD></TR>
    663663          <TR><TD>ON_STARCORE</TD>          <TD>0x00000010</TD><TD>16        </TD><TD>More than 25% of (PSF-weighted) pixels land on starcore.</TD></TR>
    … …  
    753753          <TR><TD>POOR</TD>             <TD>0x00000002</TD><TD>2              </TD><TD>Used within relphot; skip star.</TD></TR>
    754754          <TR><TD>ICRF_QSO</TD>         <TD>0x00000004</TD><TD>4              </TD><TD>object IDed with known ICRF quasar (may have ICRF position measurement)</TD></TR>
    755           <TR><TD>OTHEF_QSO</TD>        <TD>0x00000008</TD><TD>8              </TD><TD>identified as likely QSO (Hernitschek et al 2015), without ICRF reference data</TD></TR>
    756           <TR><TD>TRANSIENT</TD>        <TD>0x00000010</TD><TD>16             </TD><TD>identified as a non-periodic (stationary) transient</TD></TR>
    757           <TR><TD>VARIABLE</TD>         <TD>0x00000020</TD><TD>32             </TD><TD>identified as a periodic variable</TD></TR>
    758           <TR><TD>RRLYRA</TD>           <TD>0x00000040</TD><TD>64             </TD><TD>identified as likely RR Lyra (Hernitschek et al 2015)</TD></TR>
    759           <TR><TD>HAS_SOLSYS_DET</TD>   <TD>0x00000080</TD><TD>128            </TD><TD>identified with a known solar-system object (asteroid or other)</TD></TR>
    760           <TR><TD>ALL_SOLSYS_DET</TD>   <TD>0x00000100</TD><TD>256            </TD><TD>identified with a known solar-system object (asteroid or other)</TD></TR>
    761           <TR><TD>UNDEF_1</TD>          <TD>0x00000200</TD><TD>512            </TD><TD>Unused bit value.</TD></TR>
    762           <TR><TD>UNDEF_2</TD>          <TD>0x00000400</TD><TD>1024           </TD><TD>Unused bit value.</TD></TR>
     755          <TR><TD>HERN_QSO_P60</TD>     <TD>0x00000008</TD><TD>8              </TD><TD>identified as likely QSO (Hernitschek et al 2015), P_QSO >= 0.60</TD></TR>
     756          <TR><TD>HERN_QSO_P05</TD>     <TD>0x00000010</TD><TD>16             </TD><TD>identified as possible QSO (Hernitschek et al 2015), P_QSO >= 0.05</TD></TR>
     757          <TR><TD>HERN_RRL_P60</TD>     <TD>0x00000020</TD><TD>32             </TD><TD>identified as likely RR Lyra (Hernitschek et al 2015), P_RRLyra >= 0.60</TD></TR>
     758          <TR><TD>HERN_RRL_P05</TD>     <TD>0x00000040</TD><TD>64             </TD><TD>identified as possible RR Lyra (Hernitschek et al 2015), P_RRLyra >= 0.05</TD></TR>
     759          <TR><TD>HERN_VARIABLE</TD>    <TD>0x00000080</TD><TD>128            </TD><TD>identified as a variable based on ChiSq (Hernitschek et al 2015)</TD></TR>
     760          <TR><TD>TRANSIENT</TD>        <TD>0x00000100</TD><TD>256            </TD><TD>identified as a non-periodic (stationary) transient</TD></TR>
     761          <TR><TD>HAS_SOLSYS_DET</TD>   <TD>0x00000200</TD><TD>512            </TD><TD>at least one detection identified with a known solar-system object (asteroid or other).</TD></TR>
     762          <TR><TD>MOST_SOLSYS_DET</TD>  <TD>0x00000400</TD><TD>1024           </TD><TD>most detections identified with a known solar-system object (asteroid or other).</TD></TR>
    763763          <TR><TD>LARGE_PM</TD>         <TD>0x00000800</TD><TD>2048           </TD><TD>star with large proper motion</TD></TR>
    764764          <TR><TD>RAW_AVE</TD>          <TD>0x00001000</TD><TD>4096           </TD><TD>simple weighted average position was used (no IRLS fitting)</TD></TR>
    … …  
    777777          <TR><TD>GOOD</TD>             <TD>0x02000000</TD><TD>33554432       </TD><TD>good-quality measurement in our data (eg,PS)</TD></TR>
    778778          <TR><TD>GOOD_ALT</TD>         <TD>0x04000000</TD><TD>67108864       </TD><TD>good-quality measurement in  external data (eg, 2MASS)</TD></TR>
    779           <TR><TD>GOOD_STACK</TD>       <TD>0x08000000</TD><TD>134217728      </TD><TD>good-quality object in the stack (> 1 good stack)</TD></TR>
    780           <TR><TD>BEST_STACK</TD>       <TD>0x10000000</TD><TD>268435456      </TD><TD>the primary stack measurement are the best measurements</TD></TR>
    781           <TR><TD>SUSPECT_STACK</TD>    <TD>0x20000000</TD><TD>536870912      </TD><TD>suspect object in the stack (> 1 good or suspect stack, less than 2 good)</TD></TR>
    782           <TR><TD>BAD_STACK</TD>        <TD>0x40000000</TD><TD>1073741824     </TD><TD>good-quality object in the stack (> 1 good stack)</TD></TR>
     779          <TR><TD>GOOD_STACK</TD>       <TD>0x08000000</TD><TD>134217728      </TD><TD>good-quality object in the stack (> 1 good stack measurement)</TD></TR>
     780          <TR><TD>BEST_STACK</TD>       <TD>0x10000000</TD><TD>268435456      </TD><TD>the primary stack measurements are the best measurements</TD></TR>
     781          <TR><TD>SUSPECT_STACK</TD>    <TD>0x20000000</TD><TD>536870912      </TD><TD>suspect object in the stack (no more than 1 good measurement, 2 or more suspect or good stack measurement)</TD></TR>
     782          <TR><TD>BAD_STACK</TD>        <TD>0x40000000</TD><TD>1073741824     </TD><TD>poor-quality stack object (no more than 1 good or suspect measurement)</TD></TR>
    783783        </TABLEDATA>
    784784      </DATA>
    … …  
    809809          <TR><TD>SECF_USE_UBERCAL</TD>  <TD>0x00000008</TD><TD>8              </TD><TD>Ubercal photometry used in average measurement.</TD></TR>
    810810          <TR><TD>SECF_HAS_PS1</TD>      <TD>0x00000010</TD><TD>16             </TD><TD>PS1 photometry used in average measurement.</TD></TR>
    811           <TR><TD>SECF_HAS_STACK</TD>    <TD>0x00000020</TD><TD>32             </TD><TD>PS1 stack photometry exists.</TD></TR>
     811          <TR><TD>SECF_HAS_PS1_STACK</TD><TD>0x00000020</TD><TD>32             </TD><TD>PS1 stack photometry exists.</TD></TR>
    812812
    813813          <TR><TD>SECF_HAS_TYCHO</TD>    <TD>0x00000040</TD><TD>64             </TD><TD>Tycho photometry used for synthetic magnitudes.</TD></TR>
    814814          <TR><TD>SECF_FIX_SYNTH</TD>    <TD>0x00000080</TD><TD>128            </TD><TD>Synthetic magnitudes repaired with zeropoint map.</TD></TR>
    815815
    816           <TR><TD>PHOTOM_PASS_0</TD>     <TD>0x00000100</TD><TD>256            </TD><TD>Average magnitude calculated in 0th pass.</TD></TR>
    817           <TR><TD>PHOTOM_PASS_1</TD>     <TD>0x00000200</TD><TD>512            </TD><TD>Average magnitude calculated in 1th pass.</TD></TR>
    818           <TR><TD>PHOTOM_PASS_2</TD>     <TD>0x00000400</TD><TD>1024           </TD><TD>Average magnitude calculated in 2th pass.</TD></TR>
    819           <TR><TD>PHOTOM_PASS_3</TD>     <TD>0x00000800</TD><TD>2048           </TD><TD>Average magnitude calculated in 3th pass.</TD></TR>
    820           <TR><TD>PHOTOM_PASS_4</TD>     <TD>0x00001000</TD><TD>4096           </TD><TD>Average magnitude calculated in 4th pass.</TD></TR>
    821           <TR><TD>PSPS_OBJ_EXT</TD>      <TD>0x00002000</TD><TD>8192           </TD><TD>Extended in this band (PSPS only).</TD></TR>
     816          <TR><TD>SECF_RANK_0</TD>       <TD>0x00000100</TD><TD>256            </TD><TD>Average magnitude uses only rank 0 detections.</TD></TR>
     817          <TR><TD>SECF_RANK_1</TD>       <TD>0x00000200</TD><TD>512            </TD><TD>Average magnitude uses only rank 1 detections.</TD></TR>
     818          <TR><TD>SECF_RANK_2</TD>       <TD>0x00000400</TD><TD>1024           </TD><TD>Average magnitude uses only rank 2 detections.</TD></TR>
     819          <TR><TD>SECF_RANK_3</TD>       <TD>0x00000800</TD><TD>2048           </TD><TD>Average magnitude uses only rank 3 detections.</TD></TR>
     820          <TR><TD>SECF_RANK_4</TD>       <TD>0x00001000</TD><TD>4096           </TD><TD>Average magnitude uses only rank 4 detections.</TD></TR>
    822821          <TR><TD>SECF_STACK_PRIMARY</TD><TD>0x00004000</TD><TD>16384          </TD><TD>PS1 stack photometry comes from primary skycell.</TD></TR>
     822          <TR><TD>SECF_OBJ_EXT</TD>      <TD>0x01000000</TD><TD>16777216       </TD><TD>Extended in this band.</TD></TR>
    823823        </TABLEDATA>
    824824      </DATA>
    … …  
    847847          <TR><TD>QF_OBJ_GOOD</TD>         <TD>0x00000004</TD><TD>4       </TD><TD>Good-quality measurement in our data (eg,PS).</TD></TR>
    848848          <TR><TD>QF_OBJ_GOOD_ALT</TD>     <TD>0x00000008</TD><TD>8       </TD><TD>Good-quality measurement in  external data (eg, 2MASS).</TD></TR>
    849           <TR><TD>QF_OBJ_GOOD_STACK</TD>   <TD>0x00000010</TD><TD>16      </TD><TD>Good-quality object in the stack (> 1 good stack).</TD></TR>
    850           <TR><TD>QF_OBJ_SUSPECT_STACK</TD><TD>0x00000020</TD><TD>32      </TD><TD>Suspect object in the stack (> 1 good or suspect stack, less tham 2 good).</TD></TR>
    851           <TR><TD>QF_OBJ_BAD_STACK</TD>    <TD>0x00000040</TD><TD>64      </TD><TD>Good-quality object in the stack (> 1 good stack).</TD></TR>
     849          <TR><TD>QF_OBJ_GOOD_STACK</TD>   <TD>0x00000010</TD><TD>16      </TD><TD>good-quality object in the stack (> 1 good stack measurement)</TD></TR>
     850          <TR><TD>QF_OBJ_BEST_STACK</TD>   <TD>0x00000020</TD><TD>32      </TD><TD>the primary stack measurements are the best measurements.</TD></TR>
     851          <TR><TD>QF_OBJ_SUSPECT_STACK</TD><TD>0x00000040</TD><TD>64      </TD><TD>suspect object in the stack (no more than 1 good measurement, 2 or more suspect or good stack measurement).</TD></TR>
     852          <TR><TD>QF_OBJ_BAD_STACK</TD>    <TD>0x00000080</TD><TD>64      </TD><TD>poor-quality stack object (no more than 1 good or suspect measurement).</TD></TR>
    852853        </TABLEDATA>
    853854      </DATA>

  • trunk/ippToPsps/jython/objectbatch.py

    r39255 r39565  
    147147            # find Nsecfilt (or save in the db with dvopsps)
    148148
    149             # the math below depends on filterCount = Nsecfilt and MeanObject.row being 1 counting but cps being 0 counting?
    150             # cps.row has a count of MeanObject.row * Nsecfilt + Nfilter
    151             #  " + cpsTable + " AS cps ON (cps.row = (MeanObject.row* " + str(filterCount) + ")-(" + str(filterCount) + " - " + str(filter[0]) + ")) \
     149            # the math below depends on filterCount = Nsecfilt and
     150            # both MeanObject.row and cps.row being 1 counting, and
     151            # filter[0] also being 1 counting
     152
     153            # cps.row - 1 = (cpt.row - 1)*Nsecfilt + (filter[0] - 1)
     154            # cps.row     = cpt.row * Nsecfilt - (Nsecfilt - filter[0])
     155            # cps.row     = cpt.row * Nsecfilt - Nsecfilt + filter[0]
     156            # [note that the sql below has parenthesis around (Nsecfilt - filter[0])
    152157
    153158            sql = "UPDATE MeanObject JOIN \
    … …  
    169174                   ,MeanObject." + filter[1] + "MeanApMagStd   = MAG_AP_STDEV \
    170175                   ,MeanObject." + filter[1] + "MeanApMagNpt   = NUSED_AP \
    171                    ,MeanObject." + filter[1] + "Flags = (0x7fff & FLAGS) | ((FLAGS >> 11) & 0x2000) "
    172 
     176                   ,MeanObject." + filter[1] + "Flags          = FLAGS "
    173177
    174178            try: self.scratchDb.execute(sql)
    … …  
    301305        sqlLine.group("processingVersion",     "'" + str(self.skychunk.processingVersion) + "'")
    302306        sqlLine.group("objInfoFlag",     "FLAGS")
    303         sqlLine.group("qualityFlag",     "FLAGS >> 24 & 0xFF")
     307        sqlLine.group("qualityFlag",     "FLAGS >> 23 & 0xFF")
    304308        sqlLine.group("raStack",         "RA_STK")
    305309        sqlLine.group("decStack",        "DEC_STK")
    … …  
    339343        self.updateObjectThinFromCps(cpsTableName)
    340344
    341         # XXX EAM 20140724 : is this necessary??
    342         #objects can have out of range ra dec in dvo - need to find and kill them at the end
    343 
    344         self.logger.infoPair("Determining", "ra/dec range")
    345 
    346         raMin = self.scratchDb.getFromdvoSkyTable("R_MIN",self.region)
    347         raMax = self.scratchDb.getFromdvoSkyTable("R_MAX",self.region)
    348         decMin = self.scratchDb.getFromdvoSkyTable("D_MIN",self.region)
    349         decMax = self.scratchDb.getFromdvoSkyTable("D_MAX",self.region)
    350 
    351         self.logger.infoPair("R_MIN", raMin)
    352         self.logger.infoPair("R_MAX", raMax)
    353         self.logger.infoPair("D_MIN", decMin)
    354         self.logger.infoPair("D_MAX", decMax)
    355         #count out of range
    356 
    357         sql = "SELECT count(*) FROM ObjectThin where \
    358               ObjectThin.decMean > " + str(decMax) + " \
    359               or ObjectThin.decMean < " + str(decMin) + " \
    360               or ObjectThin.raMean > " + str(raMax) + " \
    361               or ObjectThin.raMean < " + str(raMin)       
    362    
    363         rs = self.scratchDb.executeQuery(sql)
    364        
    365         rs.first()
    366         nToDelete = rs.getInt(1)
    367        
    368         #delete out of range
    369        
    370  
    371         sql = "DELETE FROM ObjectThin where \
    372               ObjectThin.decMean > (" + str(decMax) + " + .0033) or \
    373               ObjectThin.decMean < (" + str(decMin) + " - .0033) or \
    374               ObjectThin.raMean > (" + str(raMax) + " + .0033) or \
    375               ObjectThin.raMean < (" + str(raMin) + " - .0033)"
    376         self.logger.infoPair("Deleting", str(nToDelete) + " objects outside of ra/dec range")
    377 
    378         try:
    379             self.scratchDb.execute(sql)
    380         except:
    381             self.logger.errorPair("Couldn't cull outsiders from ObjectThin table", sql)
    382             raise
    383 
    384         ##Don't do this till after MeanObject
    385         ##self.dvoObjects.purgeRegion(self.region)
     345        if False:
     346            # this chunk of code deletes objects which are out of ra,dec range for the table.
     347            # this was a problem in an early version of DVO for cases where the astrometry went insane.
     348            # this causes problems for the ra = 0,360 boundary (the test below does not handle that situation)
     349            # and the restrictions below are poorly defined for the regions near the pole.
     350           
     351            # in any case, ObjectThin needs to maintain the same order
     352            # as the cpt table until MeanObjects have been created or
     353            # the join to the cps table will fail
     354
     355            # XXX EAM 20140724 : is this necessary??
     356            # objects can have out of range ra dec in dvo - need to find and kill them at the end
     357
     358            self.logger.infoPair("Determining", "ra/dec range")
     359   
     360            raMin = self.scratchDb.getFromdvoSkyTable("R_MIN",self.region)
     361            raMax = self.scratchDb.getFromdvoSkyTable("R_MAX",self.region)
     362            decMin = self.scratchDb.getFromdvoSkyTable("D_MIN",self.region)
     363            decMax = self.scratchDb.getFromdvoSkyTable("D_MAX",self.region)
     364   
     365            self.logger.infoPair("R_MIN", raMin)
     366            self.logger.infoPair("R_MAX", raMax)
     367            self.logger.infoPair("D_MIN", decMin)
     368            self.logger.infoPair("D_MAX", decMax)
     369            #count out of range
     370   
     371            sql = "SELECT count(*) FROM ObjectThin where \
     372                  ObjectThin.decMean > " + str(decMax) + " \
     373                  or ObjectThin.decMean < " + str(decMin) + " \
     374                  or ObjectThin.raMean > " + str(raMax) + " \
     375                  or ObjectThin.raMean < " + str(raMin)       
     376       
     377            rs = self.scratchDb.executeQuery(sql)
     378           
     379            rs.first()
     380            nToDelete = rs.getInt(1)
     381           
     382            #delete out of range
     383           
     384     
     385            sql = "DELETE FROM ObjectThin where \
     386                  ObjectThin.decMean > (" + str(decMax) + " + .0033) or \
     387                  ObjectThin.decMean < (" + str(decMin) + " - .0033) or \
     388                  ObjectThin.raMean > (" + str(raMax) + " + .0033) or \
     389                  ObjectThin.raMean < (" + str(raMin) + " - .0033)"
     390            self.logger.infoPair("Deleting", str(nToDelete) + " objects outside of ra/dec range")
     391   
     392            try:
     393                self.scratchDb.execute(sql)
     394            except:
     395                self.logger.errorPair("Couldn't cull outsiders from ObjectThin table", sql)
     396                raise
     397
    386398        self.logger.infoPair("updatePspsUniqueIDs","start")
    387399        self.updatePspsUniqueIDs()
    388400        self.logger.infoPair("updatePspsUniqueIDs","end")
     401
    389402        self.logger.infoPair("Dropping row column from", "ObjectThin table")
    390403        self.scratchDb.dropColumn("ObjectThin", "row")
    391         ##self.logger.infoPair("Purging from scratch Db", self.region + " region")
    392404        self.logger.infoPair("Dropped row column", "objectThin")
    393         ##Don't do this till after MeanObject
    394405
    395406        self.setMinMaxObjID(["ObjectThin"])
    … …  
    415426        self.scratchDb.addRowCountColumn("MeanObject", "row")
    416427
    417 
    418 
    419        
    420 
    421         ##self.scratchDb.addRowCountColumn(cpsTableName, "row")
    422428        self.logger.infoPair("update MeanObjects from ","cps table")
    423429        self.updateMeanObjectFromCps(cpsTableName)
    … …  
    434440
    435441        if not self.populateObjectThinTable(): return False
    436         #if not self.populateObjectCalColorTable(): return False
    437442        if not self.populateMeanObjectTable(): return False
    438443
    439444        # now remove the objID duplicates. We could not do this before as cpt/cps tables relate by row number
     445
     446        ### XXX the code below first removes duplicate objID entries
     447        ### from ObjectThin, then does the same for MeanObject.  This
     448        ### is a big problem: we have no guarantee that the surviving
     449        ### rows are the correct matched rows.
     450
     451        ### force objID uniqueness on *** ObjectThin ***
    440452        self.logger.infoPair("Forcing uniqueness on", "objID in ObjectThin table")
    441453        rowCountBefore = self.scratchDb.getRowCount("ObjectThin")
    … …  
    457469        self.logger.infoPair("Number of duplicated objIDs removed", "%d out of %d" % ((rowCountBefore - rowCountAfter), rowCountBefore))
    458470
     471        ### force objID uniqueness on *** MeanObject ***
    459472        self.logger.infoPair("Forcing uniqueness on", "objID in MeanObject table")
    460473        rowCountBefore = self.scratchDb.getRowCount("MeanObject")
    … …  
    476489        self.logger.infoPair("Number of duplicated objIDs removed", "%d out of %d" % ((rowCountBefore - rowCountAfter), rowCountBefore))
    477490
     491        # delete the cpt, cps tables from the scratch mysql (or we will run out of space)
    478492        self.dvoObjects.purgeRegion(self.region)
    479493
    480         #this is abuse of something but this is how I get the object batches to crash to further investigate them
    481        
    482 #        rowCountAfter = self.scratchDb.getRowCount("Object")
    483         return True
    484 #        return False
     494        return True
Note: See TracChangeset for help on using the changeset viewer.