IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 37917


Ignore:
Timestamp:
Feb 20, 2015, 1:19:35 PM (11 years ago)
Author:
heather
Message:

current state of ipptopsps / sas37 - ob/p2/st/df work

Location:
trunk/ippToPsps
Files:
18 edited

Legend:

Unmodified
Added
Removed

  • trunk/ippToPsps/config/ippToPspsDbSchema.sql

    r37739 r37917  
    221221-- Table structure for table `pending`
    222222--
     223-- i had to do this to get uniqueness prob sorted out:
     224
     225--show index from pending;
     226--alter table pending drop index `name`;
     227--where name was broken becaue it only indedxed stage idg
    223228
    224229DROP TABLE IF EXISTS `pending`;

  • trunk/ippToPsps/config/tables.ST.vot

    r37722 r37917  
    10801080        <DESCRIPTION>Flux inside r = 3</DESCRIPTION>
    10811081      </FIELD>
    1082       <FIELD name="gcflxR3Err" arraysize="1" datatype="float" unit="Jy" default="-999">
     1082      <FIELD name="gc8flxR3Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    10831083        <DESCRIPTION>estimated error in c8flxR3</DESCRIPTION>
    10841084      </FIELD>
    … …  
    28472847        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    28482848      </FIELD>
    2849       <FIELD name="gflxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    2850         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    2851       </FIELD>
    2852       <FIELD name="gflxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    2853         <DESCRIPTION>estimated error is flxR10</DESCRIPTION>
    2854       </FIELD>
    2855       <FIELD name="gflxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    2856         <DESCRIPTION>estimated standard deviation in flxR10</DESCRIPTION>
    2857       </FIELD>
    2858       <FIELD name="gflxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    2859         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    2860       </FIELD>
    2861 
    28622849      <!-- r-band values -->
    28632850      <FIELD name="rippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    29782965        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    29792966      </FIELD>
    2980       <FIELD name="rflxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    2981         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    2982       </FIELD>
    2983       <FIELD name="rflxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    2984         <DESCRIPTION>estimated error is flxR10</DESCRIPTION>
    2985       </FIELD>
    2986       <FIELD name="rflxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    2987         <DESCRIPTION>estimated standard deviation in flxR10</DESCRIPTION>
    2988       </FIELD>
    2989       <FIELD name="rflxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    2990         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    2991       </FIELD>
    2992 
    29932967      <!-- i-band values -->
    29942968      <FIELD name="iippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    31093083        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    31103084      </FIELD>
    3111       <FIELD name="iflxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    3112         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    3113       </FIELD>
    3114       <FIELD name="iflxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    3115         <DESCRIPTION>estimated error is flxR10</DESCRIPTION>
    3116       </FIELD>
    3117       <FIELD name="iflxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    3118         <DESCRIPTION>estimated standard deviation in flxR10</DESCRIPTION>
    3119       </FIELD>
    3120       <FIELD name="iflxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    3121         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    3122       </FIELD>
    3123 
    31243085      <!-- z-band values -->
    31253086      <FIELD name="zippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    32403201        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    32413202      </FIELD>
    3242       <FIELD name="zflxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    3243         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    3244       </FIELD>
    3245       <FIELD name="zflxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    3246         <DESCRIPTION>estimated error is flxR10</DESCRIPTION>
    3247       </FIELD>
    3248       <FIELD name="zflxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    3249         <DESCRIPTION>estimated standard deviation in flxR10</DESCRIPTION>
    3250       </FIELD>
    3251       <FIELD name="zflxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    3252         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    3253       </FIELD>
    3254 
    32553203      <!-- y-band values -->
    32563204      <FIELD name="yippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    33713319        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    33723320      </FIELD>
    3373       <FIELD name="yflxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    3374         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    3375       </FIELD>
    3376       <FIELD name="yflxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    3377         <DESCRIPTION>estimated error is flxR10</DESCRIPTION>
    3378       </FIELD>
    3379       <FIELD name="yflxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    3380         <DESCRIPTION>estimated standard deviation in flxR10</DESCRIPTION>
    3381       </FIELD>
    3382       <FIELD name="yflxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    3383         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    3384       </FIELD>
    3385 
    33863321      <DATA>
    33873322        <TABLEDATA></TABLEDATA>
    … …  
    35333468        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    35343469      </FIELD>
    3535       <FIELD name="gc6flxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    3536         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    3537       </FIELD>
    3538       <FIELD name="gc6flxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    3539         <DESCRIPTION>estimated error is c6flxR10</DESCRIPTION>
    3540       </FIELD>
    3541       <FIELD name="gc6flxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    3542         <DESCRIPTION>estimated standard deviation in c6flxR10</DESCRIPTION>
    3543       </FIELD>
    3544       <FIELD name="gc6flxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    3545         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    3546       </FIELD>
    3547 
    35483470      <!-- r-band values -->
    35493471      <FIELD name="rippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    36643586        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    36653587      </FIELD>
    3666       <FIELD name="rc6flxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    3667         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    3668       </FIELD>
    3669       <FIELD name="rc6flxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    3670         <DESCRIPTION>estimated error is c6flxR10</DESCRIPTION>
    3671       </FIELD>
    3672       <FIELD name="rc6flxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    3673         <DESCRIPTION>estimated standard deviation in c6flxR10</DESCRIPTION>
    3674       </FIELD>
    3675       <FIELD name="rc6flxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    3676         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    3677       </FIELD>
    3678 
    36793588      <!-- i-band values -->
    36803589      <FIELD name="iippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    37953704        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    37963705      </FIELD>
    3797       <FIELD name="ic6flxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    3798         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    3799       </FIELD>
    3800       <FIELD name="ic6flxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    3801         <DESCRIPTION>estimated error is c6flxR10</DESCRIPTION>
    3802       </FIELD>
    3803       <FIELD name="ic6flxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    3804         <DESCRIPTION>estimated standard deviation in c6flxR10</DESCRIPTION>
    3805       </FIELD>
    3806       <FIELD name="ic6flxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    3807         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    3808       </FIELD>
    3809 
    38103706      <!-- z-band values -->
    38113707      <FIELD name="zippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    39263822        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    39273823      </FIELD>
    3928       <FIELD name="zc6flxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    3929         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    3930       </FIELD>
    3931       <FIELD name="zc6flxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    3932         <DESCRIPTION>estimated error is c6flxR10</DESCRIPTION>
    3933       </FIELD>
    3934       <FIELD name="zc6flxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    3935         <DESCRIPTION>estimated standard deviation in c6flxR10</DESCRIPTION>
    3936       </FIELD>
    3937       <FIELD name="zc6flxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    3938         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    3939       </FIELD>
    3940 
    39413824      <!-- y-band values -->
    39423825      <FIELD name="yippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    40573940        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    40583941      </FIELD>
    4059       <FIELD name="yc6flxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    4060         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    4061       </FIELD>
    4062       <FIELD name="yc6flxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    4063         <DESCRIPTION>estimated error is c6flxR10</DESCRIPTION>
    4064       </FIELD>
    4065       <FIELD name="yc6flxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    4066         <DESCRIPTION>estimated standard deviation in c6flxR10</DESCRIPTION>
    4067       </FIELD>
    4068       <FIELD name="yc6flxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    4069         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    4070       </FIELD>
    4071 
    40723942      <DATA>
    40733943        <TABLEDATA></TABLEDATA>
    … …  
    42194089        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    42204090      </FIELD>
    4221       <FIELD name="gc8flxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    4222         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    4223       </FIELD>
    4224       <FIELD name="gc8flxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    4225         <DESCRIPTION>estimated error is c8flxR10</DESCRIPTION>
    4226       </FIELD>
    4227       <FIELD name="gc8flxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    4228         <DESCRIPTION>estimated standard deviation in c8flxR10</DESCRIPTION>
    4229       </FIELD>
    4230       <FIELD name="gc8flxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    4231         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    4232       </FIELD>
    4233 
    42344091      <!-- r-band values -->
    42354092      <FIELD name="rippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    43504207        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    43514208      </FIELD>
    4352       <FIELD name="rc8flxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    4353         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    4354       </FIELD>
    4355       <FIELD name="rc8flxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    4356         <DESCRIPTION>estimated error is c8flxR10</DESCRIPTION>
    4357       </FIELD>
    4358       <FIELD name="rc8flxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    4359         <DESCRIPTION>estimated standard deviation in c8flxR10</DESCRIPTION>
    4360       </FIELD>
    4361       <FIELD name="rc8flxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    4362         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    4363       </FIELD>
    4364 
    43654209      <!-- i-band values -->
    43664210      <FIELD name="iippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    44814325        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    44824326      </FIELD>
    4483       <FIELD name="ic8flxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    4484         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    4485       </FIELD>
    4486       <FIELD name="ic8flxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    4487         <DESCRIPTION>estimated error is c8flxR10</DESCRIPTION>
    4488       </FIELD>
    4489       <FIELD name="ic8flxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    4490         <DESCRIPTION>estimated standard deviation in c8flxR10</DESCRIPTION>
    4491       </FIELD>
    4492       <FIELD name="ic8flxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    4493         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    4494       </FIELD>
    4495 
    44964327      <!-- z-band values -->
    44974328      <FIELD name="zippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    46124443        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    46134444      </FIELD>
    4614       <FIELD name="zc8flxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    4615         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    4616       </FIELD>
    4617       <FIELD name="zc8flxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    4618         <DESCRIPTION>estimated error is c8flxR10</DESCRIPTION>
    4619       </FIELD>
    4620       <FIELD name="zc8flxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    4621         <DESCRIPTION>estimated standard deviation in c8flxR10</DESCRIPTION>
    4622       </FIELD>
    4623       <FIELD name="zc8flxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    4624         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    4625       </FIELD>
    4626 
    46274445      <!-- y-band values -->
    46284446      <FIELD name="yippDetectID" arraysize="1" datatype="long" unit="dimensionless" default="NA">
    … …  
    47434561        <DESCRIPTION>estimated filling factor at R9</DESCRIPTION>
    47444562      </FIELD>
    4745       <FIELD name="yc8flxR10" arraysize="1" datatype="float" unit="Jy" default="-999">
    4746         <DESCRIPTION>Flux inside r = 10</DESCRIPTION>
    4747       </FIELD>
    4748       <FIELD name="yc8flxR10Err" arraysize="1" datatype="float" unit="Jy" default="-999">
    4749         <DESCRIPTION>estimated error is c8flxR10</DESCRIPTION>
    4750       </FIELD>
    4751       <FIELD name="yc8flxR10Std" arraysize="1" datatype="float" unit="Jy" default="-999">
    4752         <DESCRIPTION>estimated standard deviation in c8flxR10</DESCRIPTION>
    4753       </FIELD>
    4754       <FIELD name="yc8flxR10Fill" arraysize="1" datatype="float" unit="dimensionless" default="-999">
    4755         <DESCRIPTION>estimated filling factor at R10</DESCRIPTION>
    4756       </FIELD>
    4757 
    47584563      <DATA>
    47594564        <TABLEDATA></TABLEDATA>

  • trunk/ippToPsps/jython/detectionbatch.py

    r37763 r37917  
    611611
    612612    def updatePspsUniqueIDs(self,table, ota):
    613         sql = "UPDATE "+table+" join (select @r:=@r+1 rownum, objID from \
    614         (select @r:=0) r, "+table+" t) as foo using (objID) set \
    615         uniquePspsP2id = ((" +str(self.batchID)+ "*1000000000 ) \
    616         + (" +str(ota) + "*10000000  )+rownum)"
     613        #sql = "UPDATE "+table+" join (select @r:=@r+1 rownum, objID from \
     614        #(select @r:=0) r, "+table+" t) as foo using (objID) set \
     615        #uniquePspsP2id = ((" +str(self.batchID)+ "*1000000000 ) \
     616        #+ (" +str(ota) + "*10000000  )+rownum)"
     617        sql = "UPDATE "+table+" set uniquePspsP2id = (("+str(self.batchID)+"*1000000000 ) + (" +str(ota) + "*10000000  )+ row)"
    617618        try: self.scratchDb.execute(sql)
    618619        except:
    … …  
    647648        #self.logger.info("successful populate ");
    648649        # now add DVO IDs
     650        self.logger.infoPair("Adding 'row' column to ", "Detection_"+chipname)
     651        #detection has a primary index, so don't add, use this one
     652        self.scratchDb.addRowCountColumnNoKey("Detection_"+chipname, "row")
    649653        self.updateDvoIDs("Detection_" + chipname, self.imageIDs[chipname])
    650654        #self.logger.info("updated dvoids")
    651655        results['NULLOBJID'] = self.scratchDb.reportAndDeleteRowsWithNULLS("Detection_" + chipname, "objID")
    652656        #self.logger.info("deleted nulls")
     657
     658        ota = x*10+y
     659
     660
    653661        self.logger.info("add psps unique p2 ids")
    654         ota = x*10+y
    655662        self.updatePspsUniqueIDs("Detection_" + chipname, ota)
     663        self.logger.infoPair("Dropping 'row' column from ", "Detection_"+chipname)
     664        self.scratchDb.dropColumn("Detection_"+chipname,"row")
    656665        self.updateImageID("Detection_" + chipname, x, y)
    657666        #self.logger.info("updateImageId")

  • trunk/ippToPsps/jython/diffbatch.py

    r37763 r37917  
    327327        self.logger.infoPair("need to cull"," nulls objid")
    328328        results= self.scratchDb.reportAndDeleteRowsWithNULLS("DiffDetection", "diffObjID")
     329        self.logger.infoPair("Adding 'row' columns to", "DiffDetection tables")
     330        # diff has a primary key, so use this instead
     331        self.scratchDb.addRowCountColumnNoKey("DiffDetection", "row")
    329332        self.logger.infoPair("adding","uniquePSPSDFid")
    330333        self.updatePspsUniqueIDs("DiffDetection")
     334        self.logger.infoPair("Dropping row column from", "DiffDetection table")
     335        self.scratchDb.dropColumn("DiffDetection", "row")
     336 
    331337
    332338    '''
    … …  
    424430
    425431    def updatePspsUniqueIDs(self,table):
    426         sql = "UPDATE "+table+" join (select @r:=@r+1 rownum, diffobjID from \
    427         (select @r:=0) r, "+table+" t) as foo using (diffobjID) set \
    428         uniquePspsDFid = ((" +str(self.batchID)+ "*1000000000 ) + rownum)"
     432        #sql = "UPDATE "+table+" join (select @r:=@r+1 rownum, diffobjID from \
     433        #(select @r:=0) r, "+table+" t) as foo using (diffobjID) set \
     434        #uniquePspsDFid = ((" +str(self.batchID)+ "*1000000000 ) + rownum)"
     435        sql = "UPDATE DiffDetection set uniquePspsDFid = (("+str(self.batchID)+"*1000000000 ) + row)"
    429436        try: self.scratchDb.execute(sql)
    430437        except:

  • trunk/ippToPsps/jython/diffobjectbatch.py

    r37763 r37917  
    177177            sql += ")) "
    178178
    179             sql += "SET DiffDetObject.n" + filter[1] + " = NCODE, "
    180             sql += "DiffDetObject.nDetections = DiffDetObject.nDetections + cps.NSTACK_DET"
     179            sql += "SET DiffDetObject.n" + filter[1] + " = NWARP, "
     180            sql += "DiffDetObject.nDetections = DiffDetObject.nDetections + cps.NWARP"
    181181            self.logger.info(sql)
    182             self.scratchDb.execute(sql)
     182            try:
     183                self.scratchDb.execute(sql)
     184            except:
     185                self.logger.errorPair("failed sql", sql)
     186                return False
     187
    183188
    184189        # XXX this does not seem like a good thing to leave in SQL
    … …  
    187192            # now do a sum of n[filters], but do not include the ones with -999
    188193            sql  = "UPDATE DiffDetObject "
    189             sql += "SET nDetections = nDetections + n" + filter[1]
    190             sql += " WHERE n" + filter[1] + " != -999"
    191             self.scratchDb.execute(sql)
     194            sql += "SET nDetections = nDetections + " + filter[1] +"nTotal "
     195            sql += " WHERE " + filter[1] + "nTotal != -999"
     196            try:
     197                self.scratchDb.execute(sql)
     198            except:
     199                self.logger.errorPair("failed sql", sql)
     200                return False
     201
    192202
    193203    '''
    … …  
    368378
    369379    def updatePspsUniqueIDs(self,table):
    370         sql = "UPDATE "+table+" join (select @r:=@r+1 rownum, diffobjID from \
    371         (select @r:=0) r, "+table+" t) as foo using (diffobjID) set \
    372         uniquePspsDOid = ((" +str(self.batchID)+ "*1000000000 ) + rownum)"
     380#        sql = "UPDATE "+table+" join (select @r:=@r+1 rownum, diffobjID from \
     381#        (select @r:=0) r, "+table+" t) as foo using (diffobjID) set \
     382#        uniquePspsDOid = ((" +str(self.batchID)+ "*1000000000 ) + rownum)"
     383        sql = "UPDATE DiffDetObject set uniquePspsDOid = (("+str(self.batchID)+"*1000000000 ) + row)"
    373384        try: self.scratchDb.execute(sql)
    374385        except:
    … …  
    407418        self.scratchDb.execute("ALTER TABLE DiffDetObject CHANGE dec_ `dec` double")
    408419
     420        self.logger.infoPair("Adding 'row' columns to", "Diff Det Object")
     421        self.scratchDb.addRowCountColumn("DiffDetObject", "row")
     422        self.logger.infoPair("Adding unique", "Diff Det Object ")
    409423        self.updatePspsUniqueIDs("DiffDetObject")
     424        self.logger.infoPair("Dropping row column from", "DiffDetObject table")
     425        self.scratchDb.dropColumn("DiffDetObject", "row")
    410426
    411427        self.dvoObjects.purgeRegion(self.region)

  • trunk/ippToPsps/jython/forcedobjectbatch.py

    r37551 r37917  
    180180        for filter in filters:
    181181            # now do a sum of n[filters], but do not include the ones with -999
    182             sql  = "UPDATE ObjectThin "
    183             sql += "SET nDetections = nDetections + n" + filter[1]
     182            sql  = "UPDATE ForcedMeanObject"
     183            sql += "SET nDetections = nDetections + " + filter[1] +"nTotal "
    184184            sql += " WHERE n" + filter[1] + " != -999"
    185185            self.scratchDb.execute(sql)
     186            try: self.scratchDb.execute(sql)
     187            except:
     188                self.logger.errorPair("failed update MeanObject nDetections", sql)
     189                raise
    186190
    187191    '''

  • trunk/ippToPsps/jython/forcedwarpbatch.py

    r37592 r37917  
    224224        self.generateRandomIDs()
    225225
     226        self.logger.infoPair("Adding 'row' columns to", "ForcedWarpMeasurement")
     227        self.scratchDb.addRowCountColumn("ForcedWarpMeasurement", "row")
     228        self.logger.infoPair("Adding unique ids to", "ForcedWarpMeasurement")
    226229        self.generateUniquePspsIDs()
     230        self.logger.infoPair("Dropping row column from", "ForcedWarpMeasurement table")
     231        self.scratchDb.dropColumn("ForcedWarpMeasurement", "row")
    227232           
    228233        # add indexes ForcedWarpMeasurement
    … …  
    342347
    343348    def generatePspsUniqueIDs(self):
    344         sql = "UPDATE ForcedWarpMeasurement join (select @r:=@r+1 rownum, objID from \
    345         (select @r:=0) r, ForcedWarpMeasurement t) as foo using (objID) set \
    346         uniquePspsFWid = ((" +str(self.batchID)+ "*1000000000 ) + rownum)"
     349#        sql = "UPDATE ForcedWarpMeasurement join (select @r:=@r+1 rownum, objID from \
     350#        (select @r:=0) r, ForcedWarpMeasurement t) as foo using (objID) set \
     351#        uniquePspsFWid = ((" +str(self.batchID)+ "*1000000000 ) + rownum)"
     352        sql = "UPDATE ForcedWarpMeasurement set uniquePspsFWid = (("+str(self.batchID)+"*1000000000 ) + row)"
     353
    347354        try: self.scratchDb.execute(sql)
    348355        except:

  • trunk/ippToPsps/jython/gpc1db.py

    r37754 r37917  
    145145        elif batchType == "FW":
    146146           
    147             stage = "forcedwarp"
    148             sql = "SELECT DISTINCT sky_id, radeg, decdeg FROM fullForceInput \
    149                    JOIN addRun ON(fullForceInput.ff_id = addRun.stage_id, fullForceInput.warp_id = addRun.stage_extra1) \
     147            stage = "fullforce"
     148            sql = "SELECT DISTINCT add_id, radeg, decdeg FROM fullForceInput \
     149                   JOIN addRun ON(fullForceInput.ff_id = addRun.stage_id AND fullForceInput.warp_id = addRun.stage_extra1) \
    150150                   JOIN minidvodbRun USING(minidvodb_name) \
    151151                   JOIN minidvodbProcessed USING(minidvodb_id) \

  • trunk/ippToPsps/jython/ipptopspsdb.py

    r37748 r37917  
    13421342               (" + str(box_id) + ", '" + batchType + "', " + str(id) + ")"
    13431343
     1344            print sql
    13441345            try: self.execute(sql)
    13451346            except:
    … …  
    15511552
    15521553        # XXX EAM : need to do something special at the poles...
    1553         if (boxDim['DEC'] > 85): raise
    1554 
     1554        #if (boxDim['DEC'] > 85): raise
     1555        #heather commented out for oB   
     1556   
    15551557        # a given box centered at RA,DEC with (linear) size of SIDE has the following bounds
    15561558        dR = 0.5*boxDim['SIDE'] / math.cos(math.radians(boxDim['DEC']))

  • trunk/ippToPsps/jython/mysql.py

    r37246 r37917  
    258258            self.execute(sql)
    259259        except: pass
    260             #self.logger.warn("Index already in place on '" + column + "' for table '" + table + "'")
     260           
     261           # self.logger.warn("Index already in place on '" + column + "' for table '" + table + "'")
    261262    '''
    262263    Changes the database engine for this table to InnoDb
    … …  
    477478        self.execute("SET @i = 0")
    478479        self.execute("UPDATE " + table + " SET " + colName + "=(@i:=@i+1)")
    479         self.execute("ALTER TABLE " + table + " ADD PRIMARY KEY (" + colName + ")")
    480 
     480        self.execute("ALTER IGNORE TABLE " + table + " ADD PRIMARY KEY (" + colName + ")")
     481
     482    '''                                                                                                                                             
     483    Adds a column to this table which contains a row count                                                                                         
     484    '''
     485    def addRowCountColumnNoKey(self, table, colName):
     486
     487        self.execute("ALTER TABLE " + table + " ADD COLUMN " + colName + " INTEGER FIRST")
     488        self.execute("SET @i = 0")
     489        self.execute("UPDATE " + table + " SET " + colName + "=(@i:=@i+1)")
     490

  • trunk/ippToPsps/jython/queue.py

    r37246 r37917  
    2929        super(Queue, self).__init__(argv)
    3030
    31         # create various objects
     31       # create various objects
    3232        self.gpc1Db = Gpc1Db(self.logger, self.config)
    3333
    … …  
    132132                    #dec = self.skychunk.minDec
    133133
    134                     if (dec > 85): raise
    135 
     134                    #if (dec > 85): raise
     135                    # this may be a problem for poles # HAF XXX
     136                    if (dec > 90): raise
    136137                    # print "dec, dD: ", dec, dD ,self.skychunk.maxDec
    137138
    … …  
    170171
    171172                        dec = dec + 2*dD
    172                         if (dec > 85): raise
    173 
    174                         #dR = 0.5*self.skychunk.boxSize / math.cos(math.radians(dec))
     173                        #if (dec > 85): raise
     174                        if (dec > 90): raise
     175                       #dR = 0.5*self.skychunk.boxSize / math.cos(math.radians(dec))
    175176                        dR = 0.5*self.skychunk.boxSize
    176177                        ra = self.skychunk.minRa + dR

  • trunk/ippToPsps/jython/stackbatch.py

    r37763 r37917  
    293293        self.generateRandomIDs()
    294294
     295        self.logger.infoPair("Adding 'row' columns to", "StackObjectThin")
     296        self.scratchDb.addRowCountColumn("StackObjectThin", "row")
     297        self.logger.infoPair("Adding unique psps ids to", "StackObjectThin")
    295298        self.generateStackDetectRowIDsAndPspsUniqueIDs()
     299        self.logger.infoPair("Dropping row column from", "StackObjectThin table")
     300        self.scratchDb.dropColumn("StackObjectThin", "row")
     301
    296302
    297303        # add indexes StackObjectThin
    … …  
    372378                self.logger.errorPair('failed sql: ', sql)
    373379                raise
     380
     381
     382            # we do not yet set bestdetection, this is a placeholder until we do
     383            sql = "UPDATE StackObjectThin set bestDetection = 0"
     384            try:
     385                self.scratchDb.execute(sql)
     386            except:
     387                self.logger.errorPair('failed sql: ', sql)
     388                raise
     389
    374390
    375391        self.tablesToExport.append("StackObjectThin")
    … …  
    756772
    757773        minRadius = 1
    758         maxRadius = 10
     774        maxRadius = 9
    759775        if tablename == 'StackApFlx':
    760776            minRadius = 3
    … …  
    765781
    766782        sqlLine = sqlUtility("UPDATE " + tablename + " SET")
    767         for radius in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]:
     783        for radius in [1, 2, 3, 4, 5, 6, 7, 8, 9]:
    768784            if radius < minRadius: continue
    769785            if radius > maxRadius: continue
    … …  
    950966
    951967    def generateStackDetectRowIDsAndPspsUniqueIDs(self):
    952         sql = "UPDATE StackObjectThin join (select @r:=@r+1 rownum, objID from \
    953         (select @r:=0) r, StackObjectThin t) as foo using (objID) set \
    954         stackDetectRowID = ((" + str(self.batchID) + " << 32 ) + rownum), \
    955         uniquePspsSTid = ((" +str(self.batchID)+ "*1000000000 ) + rownum)";
     968#        sql = "UPDATE StackObjectThin join (select @r:=@r+1 rownum, objID from \
     969#        (select @r:=0) r, StackObjectThin t) as foo using (objID) set \
     970#        stackDetectRowID = ((" + str(self.batchID) + " << 32 ) + rownum), \
     971#        uniquePspsSTid = ((" +str(self.batchID)+ "*1000000000 ) + rownum)";
     972        sql = "UPDATE StackObjectThin set uniquePspsSTid = (("+str(self.batchID)+"*1000000000 ) + row)"
     973
    956974        try: self.scratchDb.execute(sql)
    957975        except:
    … …  
    10101028            self.scratchDb.addColumn(table, filter + "haveData", "tinyint")
    10111029
    1012         fields = "objID, ippObjID, randomStackObjID, primaryDetection, bestDetection, uniquePspsSTID, "
     1030        fields = "objID, ippObjID, randomStackObjID, primaryDetection, bestDetection, uniquePspsSTID, StackDetectRowID, "
    10131031
    10141032        for i in range(len(self.filters)):
    … …  
    10161034            fields += filter + "ippDetectID, "
    10171035            fields += filter + "stackDetectID, "
     1036#            fields += filter + "StackDetectRowID, "
    10181037            fields += filter + "stackMetaID " # careful here: do not add comma to last entry
    10191038            if (i < len(self.filters) - 1):
    10201039                fields += ", "
    10211040
    1022         if (table == "StackObjectAttributes"):
    1023             fields += ", stackDetectRowID "
     1041        #if (table == "StackObjectAttributes"):
     1042        #    fields += ", stackDetectRowID "
    10241043        sql  = "INSERT INTO " + table + " (" + fields + ") "
    10251044        sql += "SELECT " + fields + " FROM " + mainTable

  • trunk/ippToPsps/pspsschema/SchemaNew19.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/StackApFlxEGCon6

    r37592 r37917  
    4949gc6flxR9Std     Jy              REAL    4       -999    estimated standard deviation in c1flxR9
    5050gc6flxR9Fill    dimensionless   REAL    4       -999    estimated filling factor at R9
    51 gc6flxR10       Jy              REAL    4       -999    Flux inside r = 10
    52 gc6flxR10Err    Jy              REAL    4       -999    estimated error is c1flxR10
    53 gc6flxR10Std    Jy              REAL    4       -999    estimated standard deviation in c1flxR10
    54 gc6flxR10Fill   dimensionless   REAL    4       -999    estimated filling factor at R10
    5551
    5652

  • trunk/ippToPsps/pspsschema/SchemaNew19.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/StackApFlxEGCon6.txt

    r37592 r37917  
    4949gc6flxR9Std      - cmf : APER_FLUX_STDEV_9 : ** badly defined in cmf
    5050gc6flxR9Fill     + cmf : APER_FILL_9     
    51 gc6flxR10        - cmf : APER_FLUX_10        ** does not exist in cmf
    52 gc6flxR10Err     - cmf : APER_FLUX_ERR_10    ** does not exist in cmf
    53 gc6flxR10Std     - cmf : APER_FLUX_STDEV_10  ** does not exist in cmf
    54 gc6flxR10Fill    - cmf : APER_FILL_10        ** does not exist in cmf
     51

  • trunk/ippToPsps/pspsschema/SchemaNew19.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/StackApFlxEGCon8

    r37592 r37917  
    4949gc8flxR9Std     Jy              REAL    4       -999    estimated standard deviation in c2flxR9
    5050gc8flxR9Fill    dimensionless   REAL    4       -999    estimated filling factor at R9
    51 gc8flxR10       Jy              REAL    4       -999    Flux inside r = 10
    52 gc8flxR10Err    Jy              REAL    4       -999    estimated error is c2flxR10
    53 gc8flxR10Std    Jy              REAL    4       -999    estimated standard deviation in c2flxR10
    54 gc8flxR10Fill   dimensionless   REAL    4       -999    estimated filling factor at R10
    5551
    5652-- repeat for other filters with appropriate leading prefex.

  • trunk/ippToPsps/pspsschema/SchemaNew19.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/StackApFlxEGCon8.txt

    r37592 r37917  
    4949gc8flxR9Std      - cmf : APER_FLUX_STDEV_9 : ** badly defined in cmf
    5050gc8flxR9Fill     + cmf : APER_FILL_9     
    51 gc8flxR10        - cmf : APER_FLUX_10        ** does not exist in cmf
    52 gc8flxR10Err     - cmf : APER_FLUX_ERR_10    ** does not exist in cmf
    53 gc8flxR10Std     - cmf : APER_FLUX_STDEV_10  ** does not exist in cmf
    54 gc8flxR10Fill    - cmf : APER_FILL_10        ** does not exist in cmf
     51

  • trunk/ippToPsps/pspsschema/SchemaNew19.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/StackApFlxEGUnc

    r37592 r37917  
    4949gflxR9Std       Jy              REAL    4       -999    estimated standard deviation in flxR9
    5050gflxR9Fill      dimensionless   REAL    4       -999    estimated filling factor at R9
    51 gflxR10         Jy              REAL    4       -999    Flux inside r = 10
    52 gflxR10Err      Jy              REAL    4       -999    estimated error is flxR10
    53 gflxR10Std      Jy              REAL    4       -999    estimated standard deviation in flxR10
    54 gflxR10Fill     dimensionless   REAL    4       -999    estimated filling factor at R10

  • trunk/ippToPsps/pspsschema/SchemaNew19.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/StackApFlxEGUnc.txt

    r37592 r37917  
    4949gflxR9Std        - cmf : APER_FLUX_STDEV_9 : ** badly defined in cmf
    5050gflxR9Fill       + cmf : APER_FILL_9     
    51 gflxR10          - cmf : APER_FLUX_10        ** does not exist in cmf
    52 gflxR10Err       - cmf : APER_FLUX_ERR_10    ** does not exist in cmf
    53 gflxR10Std       - cmf : APER_FLUX_STDEV_10  ** does not exist in cmf
    54 gflxR10Fill      - cmf : APER_FILL_10        ** does not exist in cmf
     51
Note: See TracChangeset for help on using the changeset viewer.