IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 37577


Ignore:
Timestamp:
Nov 10, 2014, 3:00:51 PM (12 years ago)
Author:
heather
Message:

many fixes, mostly fixing bugs pointed out by pspswg, including missing
objects, random numbers, ext_nsigma

missing objects - (things which have detections and stacks) - these are caused
when we cull objects outside the dvo range. Gene says there is a buffer of
about .09 arcmins and suggests we cull things outside of .2 arcmins. This has
been implemented

random ids are not so random due to not setting a seed for the loaders. Random
numbers for a batch are seeded with the batch_ID

requests for sgSep / ExtNSigma as we had back in the good ol' days. This has
been added back for stack

Location:
trunk/ippToPsps
Files:
8 edited

Legend:

Unmodified
Added
Removed

  • trunk/ippToPsps/config/tables.ST.vot

    r37476 r37577  
    150150      </FIELD>
    151151      <FIELD name="randomStackObjID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    152         <DESCRIPTION>random stack id</DESCRIPTION>
     152        <DESCRIPTION>random for stack detections (not the same as random objid)</DESCRIPTION>
    153153      </FIELD>
    154154      <FIELD name="stackDetectRowID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    524524        <DESCRIPTION>exposure time (informational)</DESCRIPTION>
    525525      </FIELD>
     526      <FIELD name="gExtNSigma" arraysize="1" datatype="float" unit="unknown" default="-999">
     527        <DESCRIPTION>extnsigma is a measure of how extended something is</DESCRIPTION>
     528      </FIELD>
    526529      <FIELD name="gpsfMajorFWHM" arraysize="1" datatype="float" unit="arcsec" default="-999">
    527530        <DESCRIPTION>PSF width in major axis</DESCRIPTION>
    … …  
    622625        <DESCRIPTION>exposure time (informational)</DESCRIPTION>
    623626      </FIELD>
     627      <FIELD name="rExtNSigma" arraysize="1" datatype="float"  unit="unknown" default="-999">
     628        <DESCRIPTION>extnsigma is a measure of how extended something is</DESCRIPTION>
     629      </FIELD>
    624630      <FIELD name="rpsfMajorFWHM" arraysize="1" datatype="float" unit="arcsec" default="-999">
    625631        <DESCRIPTION>PSF width in major axis</DESCRIPTION>
    … …  
    720726        <DESCRIPTION>exposure time (informational)</DESCRIPTION>
    721727      </FIELD>
     728      <FIELD name="iExtNSigma" arraysize="1" datatype="float" unit="unknown" default="-999">
     729        <DESCRIPTION>extnsigma is a measure of how extended something is</DESCRIPTION>
     730      </FIELD>
    722731      <FIELD name="ipsfMajorFWHM" arraysize="1" datatype="float" unit="arcsec" default="-999">
    723732        <DESCRIPTION>PSF width in major axis</DESCRIPTION>
    … …  
    818827        <DESCRIPTION>exposure time (informational)</DESCRIPTION>
    819828      </FIELD>
     829      <FIELD name="zExtNSigma" arraysize="1" datatype="float" unit="unknown" default="-999">
     830        <DESCRIPTION>extnsigma is a measure of how extended something is</DESCRIPTION>
     831      </FIELD>
    820832      <FIELD name="zpsfMajorFWHM" arraysize="1" datatype="float" unit="arcsec" default="-999">
    821833        <DESCRIPTION>PSF width in major axis</DESCRIPTION>
    … …  
    916928        <DESCRIPTION>exposure time (informational)</DESCRIPTION>
    917929      </FIELD>
     930      <FIELD name="yExtNSigma" arraysize="1" datatype="float" unit="unknown" default="-999">
     931        <DESCRIPTION>extnsigma is a measure of how extended something is</DESCRIPTION>
     932      </FIELD>
    918933      <FIELD name="ypsfMajorFWHM" arraysize="1" datatype="float" unit="arcsec" default="-999">
    919934        <DESCRIPTION>PSF width in major axis</DESCRIPTION>

  • trunk/ippToPsps/jython/detectionbatch.py

    r37551 r37577  
    336336
    337337        sqlLine.group("ippDetectID",     "IPP_IDET")                                               
    338         sqlLine.group("randomDetID",     "FLOOR(RAND()*9223372036854775807)")                       
     338        sqlLine.group("randomDetID",     "FLOOR(RAND("+str(self.batchID)+")*9223372036854775807)")                       
    339339        sqlLine.group("filterID",        str(self.filterID))                                       
    340340        sqlLine.group("surveyID",        str(self.surveyID))                                       

  • trunk/ippToPsps/jython/diffobjectbatch.py

    r36697 r37577  
     1#!/usr/bin/env jython
     2
     3import os.path
     4import sys
     5
     6import stilts
     7from java.lang import *
     8from java.sql import *
     9
     10from xml.etree.ElementTree import ElementTree, Element, tostring
     11
     12from batch import Batch
     13from gpc1db import Gpc1Db
     14from ipptopspsdb import IppToPspsDb
     15from scratchdb import ScratchDb
     16from dvoobjects import DvoObjects
     17from sqlUtility import sqlUtility
     18
     19import logging.config
     20
     21'''
     22DiffObjectBatch class
     23
     24This class, a sub-class of Batch, processes individual DVO region file pairs (cpt and cps files). The cpt file contains a list of all objects in that region, the cps file contains the magnitudes and so has a row per filter per object (so, generally, has rows = 5 x number of objects). The Photcode.dat table us used to figure out the number and order of filters as they appear in the cps file.
     25
     26'''
     27class DiffObjectBatch(Batch):
     28
     29    '''
     30    Constructor
     31    '''
     32    def __init__(self,
     33                 logger,
     34                 config,
     35                 skychunk,
     36                 gpc1Db,
     37                 ippToPspsDb,
     38                 scratchDb,
     39                 dvoID,
     40                 batchID):
     41
     42       super(DiffObjectBatch, self).__init__(
     43               logger,
     44               config,
     45               skychunk,
     46               gpc1Db,
     47               ippToPspsDb,
     48               scratchDb,
     49               dvoID,
     50               batchID,
     51               "OB",
     52               None)
     53
     54       try:
     55           self.dvoObjects = DvoObjects(self.logger, self.config, self.skychunk, self.ippToPspsDb, self.scratchDb)
     56       except:
     57           self.logger.errorPair("Unable to create instance of", "DvoObjects")
     58           raise
     59
     60       # create an output filename, which is {dvoINDEX}.FITS
     61       self.outputFitsFile = "%08d.FITS" % self.id
     62       self.outputFitsPath = "%s/%s" % (self.localOutPath, self.outputFitsFile)
     63
     64       # dump stuff to log
     65       self.logger.infoPair("DVO INDEX", "%d" % self.id)
     66
     67    '''
     68    Overriden from batch base-class. We import from DVO directly for objects
     69    '''
     70    def importIppTables(self, columns="*", filter=""):
     71
     72        self.region = self.scratchDb.getRegionNameFromThisDvoIndex(self.id)
     73        self.ippToPspsDb.insertObjectMeta(self.batchID, self.region)
     74        self.dvoObjects.nativeIngestRegion(self.region)
     75
     76        cptTableName = self.scratchDb.getDbFriendlyTableName(self.region + ".cpt")
     77        if not self.scratchDb.tableExists(cptTableName):
     78            return False
     79
     80        return True
     81
     82    '''
     83    Applies indexes to the PSPS tables
     84    '''
     85    def alterPspsTables(self):
     86
     87        return True
     88
     89    '''
     90    Applies indexes to the IPP tables
     91    '''
     92    def indexIppTables(self):
     93
     94        # since dvopsps is now used, no action is needed here
     95        # self.logger.infoPair("Creating indexes on", "IPP tables")
     96
     97        return True
     98
     99    '''
     100    Inserts stuff for all mags
     101    '''
     102    def updateMeanObjectFromCps(self, cpsTable):
     103
     104        # list of all filters PSPS is interested in
     105        # XXX EAM : 2014.07.24 : this list should probably be in a config file somewhere
     106        interestedFilters = ['g', 'r', 'i', 'z', 'y']
     107       
     108        filters = self.scratchDb.getOrderedListOfFiltersFromPhotcodesTable(interestedFilters)
     109   
     110        # get a count of the available filters
     111        filterCount = self.scratchDb.getCountOfFiltersFromPhotcodesTable()
     112
     113        self.logger.infoPair("Available filters in Photcodes", filters)
     114
     115        # the 'code' now defines the order in the cps file that the mags are listed for a given filter
     116        self.logger.infoPair("Adding magnitudes from", "cps table")
     117        for filter in filters:
     118
     119            filterID = self.scratchDb.getFilterID(filter[1])
     120
     121            # NOTE: Manipulation of FLAGS from cpsTable is to move ID_SECF_OBJ_EXT flag (0x01000000)
     122            # from bit 24 to bit 13 so that it fits into the SMALLINT Object.Flags
     123            # XXX EAM : 20140724 this manipulation is no longer needed : [grizy]Flags is now 4 byte (was 8 byte!)
     124
     125            # set the MeanObject fields based largely on dvopsps cps fields:
     126
     127            # XXX EAM 20140724 : filterCount is meant to match
     128            # Nsecfilt, but is potentially not determined correctly.
     129            # use a call to a dvo-native command which knows how to
     130            # find Nsecfilt (or save in the db with dvopsps)
     131
     132            # the math below depends on filterCount = Nsecfilt and MeanObject.row being 1 counting but cps being 0 counting?
     133            # cps.row has a count of MeanObject.row * Nsecfilt + Nfilter
     134            #  " + cpsTable + " AS cps ON (cps.row = (MeanObject.row* " + str(filterCount) + ")-(" + str(filterCount) + " - " + str(filter[0]) + ")) \
     135
     136            sql = "UPDATE MeanObject JOIN \
     137                   " + cpsTable + " AS cps ON (cps.row = (MeanObject.row* " + str(filterCount) + ")-(" + str(filterCount) + " - " + str(filter[0]) + ")) \
     138                   SET \
     139                    MeanObject." + filter[1] + "QfPerfect      = PSF_QF_PERF_MAX \
     140                   ,MeanObject." + filter[1] + "MeanPSFMag     = MAG \
     141                   ,MeanObject." + filter[1] + "MeanPSFMagErr  = MAG_ERR \
     142                   ,MeanObject." + filter[1] + "MeanPSFMagStd  = MAG_STDEV \
     143                   ,MeanObject." + filter[1] + "MeanPSFMagMin  = MAG_MIN \
     144                   ,MeanObject." + filter[1] + "MeanPSFMagMax  = MAG_MAX \
     145                   ,MeanObject." + filter[1] + "MeanPSFMagNpt  = NUSED \
     146                   ,MeanObject." + filter[1] + "MeanKronMag    = MAG_KRON \
     147                   ,MeanObject." + filter[1] + "MeanKronMagErr = MAG_KRON_ERR \
     148                   ,MeanObject." + filter[1] + "MeanKronMagStd = MAG_KRON_STDEV \
     149                   ,MeanObject." + filter[1] + "MeanKronMagNpt = NUSED_KRON \
     150                   ,MeanObject." + filter[1] + "MeanApMag      = MAG_AP \
     151                   ,MeanObject." + filter[1] + "MeanApMagErr   = MAG_AP_ERR \
     152                   ,MeanObject." + filter[1] + "MeanApMagStd   = MAG_AP_STDEV \
     153                   ,MeanObject." + filter[1] + "MeanApMagNpt   = NUSED_AP \
     154                   ,MeanObject." + filter[1] + "Flags = (0x7fff & FLAGS) | ((FLAGS >> 11) & 0x2000) "
     155
     156
     157            try: self.scratchDb.execute(sql)
     158            except:
     159                self.logger.errorPair("failed update MeanObject", sql)
     160                raise
     161
     162        # now set to null all MeanMagErr values > 0.5 (cut set by Gene, 2012-04-12)
     163        # XXX EAM 20140724 : keep this cut?
     164        cut = 0.5
     165        self.logger.infoPair("Setting to NULL all MeanMagErr value >", "%f" % cut)
     166        for filter in filters:
     167
     168            sql = "UPDATE MeanObject \
     169                   SET " + filter[1] + "MeanPSFMagErr = null \
     170                   WHERE " + filter[1] + "MeanPSFMagErr > " + str(cut)
     171            self.scratchDb.execute(sql)
     172
     173    '''
     174    Inserts stuff for all mags
     175    '''
     176    def updateObjectThinFromCps(self, cpsTable):
     177
     178        # list of all filters PSPS is interested in
     179        interestedFilters = ['g', 'r', 'i', 'z', 'y']
     180       
     181        filters = self.scratchDb.getOrderedListOfFiltersFromPhotcodesTable(interestedFilters)
     182   
     183        # get a count of the available filters
     184        filterCount = self.scratchDb.getCountOfFiltersFromPhotcodesTable()
     185        # filterCount = len(filters)
     186
     187        self.logger.infoPair("Available filters in Photcodes", filters)
     188
     189        # the 'code' now defines the order in the cps file that the mags are listed for a given filter
     190        self.logger.infoPair("Adding magnitudes from", "cps table")
     191        for filter in filters:
     192
     193            filterID = self.scratchDb.getFilterID(filter[1])
     194
     195            # XXX EAM 20140724 : this is quite awkward, add a objRow and ncode value to mysql db table?
     196            # sqlLine = sqlUtility()
     197
     198            sql  = "UPDATE ObjectThin JOIN "
     199            sql += cpsTable + " AS cps "
     200            sql += "ON (cps.row = (ObjectThin.row * "
     201            sql += str(filterCount) + ")-("
     202            sql += str(filterCount) + " - "
     203            sql += str(filter[0])
     204            sql += ")) "
     205
     206            sql += "SET ObjectThin.n" + filter[1] + " = NCODE, "
     207            sql += "ObjectThin.nStackDetections = ObjectThin.nStackDetections + cps.NSTACK_DET"
     208            self.logger.info(sql)
     209            self.scratchDb.execute(sql)
     210
     211        # XXX this does not seem like a good thing to leave in SQL
     212        self.logger.infoPair("Calculating nDetections from", "n[filters]")
     213        for filter in filters:
     214            # now do a sum of n[filters], but do not include the ones with -999
     215            sql  = "UPDATE ObjectThin "
     216            sql += "SET nDetections = nDetections + n" + filter[1]
     217            sql += " WHERE n" + filter[1] + " != -999"
     218            self.scratchDb.execute(sql)
     219
     220    '''
     221    give objectName to objectThin
     222    XXX EAM 20140714 : This seems quite inefficient in SQL, move to dvopsps?
     223    '''
     224    def updateObjName(self):
     225    ##this is a 4 part update:
     226        self.logger.infoPair("updating objName", "using stacks")
     227    ## if we have a stackra and dec, use stackra and dec , dec >=0
     228
     229        sql = "update ObjectThin set objName =  concat('PS1.2 J', \
     230        lpad(floor(raStack/15.),2,'0'), \
     231        lpad((raStack/15.-(floor(raStack/15.)))*60. ,2,'0'), \
     232        lpad(format(((raStack/15.-(floor(raStack/15.)))*60.-floor((raStack/15.-(floor(raStack/15.)))*60.))*60.,2),4,'0'), \
     233        '+', \
     234        lpad(floor(decStack),2,'0'), \
     235        lpad((decStack-(floor(decStack)))*60. ,2,'0'), \
     236        lpad(format(((decStack-(floor(decStack)))*60.-floor((decStack-(floor(decStack)))*60.))*60.,2),4,'0') \
     237        ) where raStack > -999 and decStack > -999 and decStack >= 0"
     238        try:
     239            self.scratchDb.execute(sql)
     240        except:
     241            self.logger.errorPair("failed sql", sql)
     242            return False
     243
     244
     245    ## if we have a stackra and dec, use stackra and dec, dec < 0
     246
     247        sql = "update ObjectThin set objName =  concat('PS1.2 J', \
     248        lpad(floor(raStack/15.),2,'0'), \
     249        lpad((raStack/15.-(floor(raStack/15.)))*60. ,2,'0'), \
     250        lpad(format(((raStack/15.-(floor(raStack/15.)))*60.-floor((raStack/15.-(floor(raStack/15.)))*60.))*60.,2),4,'0'), \
     251        '-', \
     252        lpad(floor(decStack),2,'0'), \
     253        lpad((decStack-(floor(decStack)))*60. ,2,'0'), \
     254        lpad(format(((decStack-(floor(decStack)))*60.-floor((decStack-(floor(decStack)))*60.))*60.,2),4,'0') \
     255        ) where raStack > -999 and decStack > -999 and decStack <= 0"
     256        try:
     257            self.scratchDb.execute(sql)
     258        except:
     259            self.logger.errorPair("failed sql", sql)
     260            return False
     261        self.logger.infoPair("updating objName", "using means")
     262    ## use mean ra and dec (dec >= 0)
     263        sql = "update ObjectThin set objName =  concat('PS1.2 J', \
     264        lpad(floor(raMean/15.),2,'0'), \
     265        lpad((raMean/15.-(floor(raMean/15.)))*60. ,2,'0'), \
     266        lpad(format(((raMean/15.-(floor(raMean/15.)))*60.-floor((raMean/15.-(floor(raMean/15.)))*60.))*60.,2),4,'0'), \
     267        '+', \
     268        lpad(floor(decMean),2,'0'), \
     269        lpad((decMean-(floor(decMean)))*60. ,2,'0'), \
     270        lpad(format(((decMean-(floor(decMean)))*60.-floor((decMean-(floor(decMean)))*60.))*60.,2),4,'0') \
     271        ) where raMean > -999 and decMean > -999 and decMean >= 0"
     272        try:
     273            self.scratchDb.execute(sql)
     274        except:
     275            self.logger.errorPair("failed sql", sql)
     276            return False
     277 
     278    ## use mean ra and dec (dec < 0)
     279
     280        sql = "update ObjectThin set objName =  concat('PS1.2 J', \
     281        lpad(floor(raMean/15.),2,'0'), \
     282        lpad((raMean/15.-(floor(raMean/15.)))*60. ,2,'0'), \
     283        lpad(format(((raMean/15.-(floor(raMean/15.)))*60.-floor((raMean/15.-(floor(raMean/15.)))*60.))*60.,2),4,'0'), \
     284        '-', \
     285        lpad(floor(decMean),2,'0'), \
     286        lpad((decMean-(floor(decMean)))*60. ,2,'0'), \
     287        lpad(format(((decMean-(floor(decMean)))*60.-floor((decMean-(floor(decMean)))*60.))*60.,2),4,'0') \
     288        ) where raMean > -999 and decMean > -999 and decMean < 0"
     289
     290        try:
     291            self.scratchDb.execute(sql)
     292        except:
     293            self.logger.errorPair("failed sql", sql)
     294            return False
     295
     296    '''
     297    Populates the Object table
     298    '''
     299    def populateObjectThinTable(self):
     300
     301        cptTableName = self.scratchDb.getDbFriendlyTableName(self.region + ".cpt")
     302        cpsTableName = self.scratchDb.getDbFriendlyTableName(self.region + ".cps")
     303
     304        if False:
     305            # XXX EAM 20140724 : this is probably wrong : flux measurements can be 0.0 or negative: please review
     306            self.logger.infoPair("setting to null  > 1e-38 and < 1e-38 in", "cps FLUX_KRON_ERR")
     307            sql = "UPDATE " + cpsTableName + " set FLUX_KRON_ERR = NULL where FLUX_KRON_ERR < 1e-37 AND FLUX_KRON_ERR > -1e-37 "
     308            self.exitProgram("review this code" + sql)
     309     
     310            try:
     311                self.scratchDb.execute(sql)
     312            except:
     313                self.logger.errorPair("Couldn't squash out of range stuff", sql)
     314                return False
     315
     316            self.logger.infoPair("setting to null  > 1e-38 and < 1e-38 in", "cps FLUX_KRON")
     317            sql = "UPDATE " + cpsTableName + " set FLUX_KRON = NULL where FLUX_KRON < 1e-37 AND FLUX_KRON > -1e-37 "
     318     
     319            try:
     320                self.scratchDb.execute(sql)
     321            except:
     322                self.logger.errorPair("Couldn't squash out of range stuff", sql)
     323                return False
     324
     325       
     326        self.logger.info("Populating ThinObject")
     327        self.logger.info("Inserting objects from cpt file")
     328
     329        # note "dec" is a reserved word in MySQL
     330        # XXX EAM 20140724 : do not use INGORE unless we discover unavoidable problems...
     331        # INSERT IGNORE INTO ObjectThin
     332
     333        sqlLine = sqlUtility("INSERT INTO ObjectThin (")
     334
     335        sqlLine.group("objID",           "EXT_ID")
     336        sqlLine.group("ippObjID",        "OBJ_ID + (CAT_ID << 32)") # NOTE: shift by 32 bits exactly
     337        sqlLine.group("surveyID",        "'" + str(self.surveyID) + "'")
     338        sqlLine.group("randomID",        "FLOOR(RAND("+str(self.batchID)+")*9223372036854775807)") # XXX where does this number come from??
     339        sqlLine.group("batchID",         "'" + str(self.batchID) + "'")
     340        sqlLine.group("dvoRegionID",     "CAT_ID")
     341        sqlLine.group("tessID",          "TESS_ID")
     342        sqlLine.group("projectionID",    "PROJECTION_ID")
     343        sqlLine.group("skycellID",       "SKYCELL_ID")
     344        sqlLine.group("dataRelease",     "'" + str(self.skychunk.dataRelease) + "'")
     345        sqlLine.group("objInfoFlag",     "FLAGS")
     346        sqlLine.group("qualityFlag",     "FLAGS >> 24 & 0xFF")
     347        sqlLine.group("consistencyFlag", "'0'")
     348        sqlLine.group("raStack",         "RA_STK")
     349        sqlLine.group("decStack",        "DEC_STK")
     350        sqlLine.group("raStackErr",      "RA_STK_ERR")
     351        sqlLine.group("decStackErr",     "DEC_STK_ERR")
     352        sqlLine.group("raMean",          "RA_MEAN")
     353        sqlLine.group("decMean",         "DEC_MEAN")
     354        sqlLine.group("raMeanErr",       "RA_ERR")
     355        sqlLine.group("decMeanErr",      "DEC_ERR")
     356        sqlLine.group("posMeanChisq",    "CHISQ_POS")
     357        sqlLine.group("nStackObjectRows", "'0'") # XXX I need to add / define this in dvopsps
     358        sqlLine.group("nStackDetections", "'0'")
     359        sqlLine.group("nDetections",      "'0'")
     360        sql = sqlLine.makeRaw(") SELECT ", " FROM " + cptTableName)
     361
     362        try:
     363            self.scratchDb.execute(sql)
     364        except:
     365            self.logger.errorPair("Couldn't populate Object table", sql)
     366            return False
     367 
     368        # add row count columns so we can perform joins to get colors
     369        self.logger.infoPair("Adding 'row' columns to", "Object and cps tables")
     370        self.scratchDb.addRowCountColumn("ObjectThin", "row")
     371        self.scratchDb.addRowCountColumn(cpsTableName, "row")
     372
     373##        self.insertMeanMags(cpsTableName)
     374## this is the new version
     375        self.logger.infoPair("update objName for ","objectThin")
     376        self.updateObjName()
     377
     378        self.logger.infoPair("update ObjectThin from ","cps table")
     379
     380        self.updateObjectThinFromCps(cpsTableName)
     381
     382        # XXX EAM 20140724 : is this necessary??
     383        #objects can have out of range ra dec in dvo - need to find and kill them at the end
     384
     385        self.logger.infoPair("Determining", "ra/dec range")
     386
     387        raMin = self.scratchDb.getFromdvoSkyTable("R_MIN",self.region)
     388        raMax = self.scratchDb.getFromdvoSkyTable("R_MAX",self.region)
     389        decMin = self.scratchDb.getFromdvoSkyTable("D_MIN",self.region)
     390        decMax = self.scratchDb.getFromdvoSkyTable("D_MAX",self.region)
     391
     392        self.logger.infoPair("R_MIN", raMin)
     393        self.logger.infoPair("R_MAX", raMax)
     394        self.logger.infoPair("D_MIN", decMin)
     395        self.logger.infoPair("D_MAX", decMax)
     396        #count out of range
     397
     398        sql = "SELECT count(*) FROM ObjectThin where \
     399              ObjectThin.decMean > " + str(decMax) + " \
     400              or ObjectThin.decMean < " + str(decMin) + " \
     401              or ObjectThin.raMean > " + str(raMax) + " \
     402              or ObjectThin.raMean < " + str(raMin)       
     403   
     404        rs = self.scratchDb.executeQuery(sql)
     405        rs.first()
     406        nToDelete = rs.getInt(1)
     407       
     408        #delete out of range
     409       
     410 
     411        sql = "DELETE FROM ObjectThin where \
     412              ObjectThin.decMean > (" + str(decMax) + " + .0033) or \
     413              ObjectThin.decMean < (" + str(decMin) + " - .0033) or \
     414              ObjectThin.raMean > (" + str(raMax) + " + .0033) or \
     415              ObjectThin.raMean < (" + str(raMin) + " - .0033)"
     416        self.logger.infoPair("Deleting", str(nToDelete) + " objects outside of ra/dec range")
     417
     418        try:
     419            self.scratchDb.execute(sql)
     420        except:
     421            self.logger.errorPair("Couldn't cull outsiders from ObjectThin table", sql)
     422            return False
     423
     424        self.logger.infoPair("Dropping row column from", "ObjectThin table")
     425        self.scratchDb.dropColumn("ObjectThin", "row")
     426        ##self.logger.infoPair("Purging from scratch Db", self.region + " region")
     427
     428        ##Don't do this till after MeanObject
     429        ##self.dvoObjects.purgeRegion(self.region)
     430
     431        self.setMinMaxObjID(["ObjectThin"])
     432
     433        return True
     434
     435    '''
     436    Populates the ObjectCalColor table
     437    '''
     438    def populateMeanObjectTable(self):
     439        cptTableName = self.scratchDb.getDbFriendlyTableName(self.region + ".cpt")
     440        cpsTableName = self.scratchDb.getDbFriendlyTableName(self.region + ".cps")
     441        self.logger.infoPair("Populating MeanObject from ", "ObjectThin")
     442
     443        sql = "INSERT INTO MeanObject ( objID ) SELECT objID FROM ObjectThin"
     444        try:
     445            self.scratchDb.execute(sql)
     446        except:
     447            self.logger.errorPair("Couldn't populate MeanObject table", sql)
     448            return False
     449
     450        self.logger.infoPair("Adding 'row' columns to", "MeanObject table")
     451        self.scratchDb.addRowCountColumn("MeanObject", "row")
     452 
     453
     454        ##self.scratchDb.addRowCountColumn(cpsTableName, "row")
     455        self.logger.infoPair("update MeanObjects from ","cps table")
     456        self.updateMeanObjectFromCps(cpsTableName)
     457
     458        self.logger.infoPair("Dropping row column from", "MeanObject table")
     459        self.scratchDb.dropColumn("MeanObject", "row")
     460
     461        return True
     462
     463    '''
     464    Does the processing, i.e. pulling stuff from IPP tables into PSPS tables
     465    '''
     466    def populatePspsTables(self):
     467
     468        if not self.populateObjectThinTable(): return False
     469        #if not self.populateObjectCalColorTable(): return False
     470        if not self.populateMeanObjectTable(): return False
     471
     472        # now remove the objID duplicates. We could not do this before as cpt/cps tables relate by row number
     473        self.logger.infoPair("Forcing uniqueness on", "objID in ObjectThin table")
     474        rowCountBefore = self.scratchDb.getRowCount("ObjectThin")
     475
     476        # XXX EAM : note that in mysql versions later than 5.1, this fails
     477        # unless the following is called first:
     478        # set session old_alter_table=1
     479        # follow the command with
     480        # set session old_alter_table=0
     481        # OF COURSE, this fails for mysql version < 5.5...
     482        if self.scratchDb.version > 5.1:
     483            self.scratchDb.execute("set session old_alter_table=1")
     484           
     485        self.scratchDb.execute("ALTER IGNORE TABLE ObjectThin ADD UNIQUE INDEX(objID)")
     486        if self.scratchDb.version > 5.1:
     487            self.scratchDb.execute("set session old_alter_table=0")
     488
     489        rowCountAfter = self.scratchDb.getRowCount("ObjectThin")
     490        self.logger.infoPair("Number of duplicated objIDs removed", "%d out of %d" % ((rowCountBefore - rowCountAfter), rowCountBefore))
     491
     492        self.logger.infoPair("Forcing uniqueness on", "objID in MeanObject table")
     493        rowCountBefore = self.scratchDb.getRowCount("MeanObject")
     494
     495        # XXX EAM : note that in mysql versions later than 5.1, this fails
     496        # unless the following is called first:
     497        # set session old_alter_table=1
     498        # follow the command with
     499        # set session old_alter_table=0
     500        # OF COURSE, this fails for mysql version < 5.5...
     501        if self.scratchDb.version > 5.1:
     502            self.scratchDb.execute("set session old_alter_table=1")
     503           
     504        self.scratchDb.execute("ALTER IGNORE TABLE MeanObject ADD UNIQUE INDEX(objID)")
     505        if self.scratchDb.version > 5.1:
     506            self.scratchDb.execute("set session old_alter_table=0")
     507
     508        rowCountAfter = self.scratchDb.getRowCount("MeanObject")
     509        self.logger.infoPair("Number of duplicated objIDs removed", "%d out of %d" % ((rowCountBefore - rowCountAfter), rowCountBefore))
     510
     511        self.dvoObjects.purgeRegion(self.region)
     512
     513        #this is abuse of something but this is how I get the object batches to crash to further investigate them
     514       
     515#        rowCountAfter = self.scratchDb.getRowCount("Object")
     516        return True
     517#        return False

  • trunk/ippToPsps/jython/forcedwarpbatch.py

    r37551 r37577  
    326326
    327327    def generateRandomIDs(self):
    328         sql = "UPDATE ForcedWarpMeasurement set randomWarpID = FLOOR(RAND()*9223372036854775807)";
     328           sql = "UPDATE ForcedWarpMeasurement set randomWarpID = FLOOR(RAND("+str(self.batchID)+")*9223372036854775807)";
    329329        try: self.scratchDb.execute(sql)
    330330        except:

  • trunk/ippToPsps/jython/objectbatch.py

    r37246 r37577  
    336336        sqlLine.group("ippObjID",        "OBJ_ID + (CAT_ID << 32)") # NOTE: shift by 32 bits exactly
    337337        sqlLine.group("surveyID",        "'" + str(self.surveyID) + "'")
    338         sqlLine.group("randomID",        "FLOOR(RAND()*9223372036854775807)") # XXX where does this number come from??
     338        sqlLine.group("randomID",        "FLOOR(RAND("+str(self.batchID)+")*9223372036854775807)") # XXX where does this number come from??
    339339        sqlLine.group("batchID",         "'" + str(self.batchID) + "'")
    340340        sqlLine.group("dvoRegionID",     "CAT_ID")
    … …  
    410410 
    411411        sql = "DELETE FROM ObjectThin where \
    412               ObjectThin.decMean > " + str(decMax) + " or \
    413               ObjectThin.decMean < " + str(decMin) + " or \
    414               ObjectThin.raMean > " + str(raMax) + " or \
    415               ObjectThin.raMean < " + str(raMin)
     412              ObjectThin.decMean > (" + str(decMax) + " + .0033) or \
     413              ObjectThin.decMean < (" + str(decMin) + " - .0033) or \
     414              ObjectThin.raMean > (" + str(raMax) + " + .0033) or \
     415              ObjectThin.raMean < (" + str(raMin) + " - .0033)"
    416416        self.logger.infoPair("Deleting", str(nToDelete) + " objects outside of ra/dec range")
    417417

  • trunk/ippToPsps/jython/stackbatch.py

    r37475 r37577  
    942942
    943943    def generateRandomIDs(self):
    944         sql = "UPDATE StackObjectThin set randomStackObjID = FLOOR(RAND()*9223372036854775807)";
     944        sql = "UPDATE StackObjectThin set randomStackObjID = FLOOR(RAND("+str(self.batchID)+")*9223372036854775807)";
    945945        try: self.scratchDb.execute(sql)
    946946        except:

  • trunk/ippToPsps/pspsschema/SchemaNew19.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/StackObjectAttributes

    r37474 r37577  
    3939gKronRad          arcsec          REAL    4       -999    Kron radius
    4040gexpTime          seconds         REAL    4       -999    total exposure time (informational)
     41gExtNSigma        unknown         REAL    4       -999    EXT_NSIGMA, how extended something is
    4142gzp               magnitude       REAL    4       0       zeropoint , no error given
    4243gPlateScale       arcsec/pixel    REAL    4       0       plate scale

  • trunk/ippToPsps/pspsschema/SchemaNew19.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/StackObjectAttributes.txt

    r37474 r37577  
    4040gKronRad          + cmf : MOMENTS_R1 * 2.5
    4141gexpTime          + cmf : header EXPTIME
     42gExtNsigma        + cmg : EXT_NSIGMA
    4243gzp               + dvo : zp ** clarify in dvopsps
    4344gPlateScale       + cmf : PLTSCALE
Note: See TracChangeset for help on using the changeset viewer.