IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 35063


Ignore:
Timestamp:
Jan 27, 2013, 5:58:25 AM (13 years ago)
Author:
eugene
Message:

enable test mode from command-line (-test, after jython module); add parallel dvo option; make database selection more consistent and variable on test/not-test

Location:
branches/eam_branches/ipp-20121219/ippToPsps
Files:
1 added
15 edited

Legend:

Unmodified
Added
Removed

  • branches/eam_branches/ipp-20121219/ippToPsps/config/ippToPspsDbSchema.sql

    r33753 r35063  
    122122  `queue_OB` tinyint(1) default '0',
    123123  `active` tinyint(1) default '1',
     124  `parallel` tinyint(1) default '0',
    124125  UNIQUE KEY `name` (`name`)
    125126) ENGINE=InnoDB DEFAULT CHARSET=latin1;

  • branches/eam_branches/ipp-20121219/ippToPsps/config/settings.xml

    r35058 r35063  
    88  <logPath>logs</logPath>
    99
    10   <!-- local scratch Db section -->
     10  <!-- local scratch Db section (REAL version) -->
    1111  <localdatabase>
    12     <name>ipptopsps_test_scratch</name>
    13     <host>localhost</host>
     12    <name>ipptopsps_scratch</name>
     13    <host>ipp005</host>
    1414    <user>dvo</user>
    1515    <password>dvo</password>
    1616  </localdatabase>
    1717
    18   <!-- GPC1 Db section -->
     18  <!-- GPC1 Db section (REAL version) -->
    1919  <gpc1database>
    20     <!-- TEST VERSION
    21     <name>gpc1test</name>
    22     <host>localhost</host>
    23     <user>dvo</user>
    24     <password>dvo</password> -->
    25     <!-- REAL VERSION -->
    2620    <name>gpc1</name>
    2721    <host>ippdb01</host>
    … …  
    3024  </gpc1database>
    3125
    32   <!-- ippToPsps Db section -->
     26  <!-- ippToPsps Db section (REAL version) -->
    3327  <ipptopspsdatabase>
     28    <name>ipptopsps</name>
     29    <host>localhost</host>
     30    <user>dvo</user>
     31    <password>dvo</password>
     32  </ipptopspsdatabase>
     33
     34  <!-- local scratch Db section (TEST VERSION) -->
     35  <localdatabase_test>
     36    <name>ipptopsps_test_scratch</name>
     37    <host>localhost</host>
     38    <user>dvo</user>
     39    <password>dvo</password>
     40  </localdatabase_test>
     41
     42  <!-- GPC1 Db section (TEST VERSION) -->
     43  <gpc1database_test>
     44    <name>gpc1test</name>
     45    <host>localhost</host>
     46    <user>dvo</user>
     47    <password>dvo</password>
     48  </gpc1database_test>
     49
     50  <!-- ippToPsps Db section (TEST VERSION)  -->
     51  <ipptopspsdatabase_test>
    3452    <name>ipptopsps_test</name>
    3553    <host>localhost</host>
    3654    <user>dvo</user>
    3755    <password>dvo</password>
    38   </ipptopspsdatabase>
     56  </ipptopspsdatabase_test>
    3957
    4058  <!-- Czar Db section -->

  • branches/eam_branches/ipp-20121219/ippToPsps/doc/upgrade.txt

    r35062 r35063  
     1
     22013.01.26
     3
     4I have modified the methods which create the StackDetectionCalib and
     5DetectionCalib tables to change the sql from "INSERT / UPDATE" to
     6"SELECT / LOAD" operations.  these need a target tmp directory
     7different from /tmp.  the trick is that /tmp has permissions which
     8prevent my (joe-user) from removing the temp files in the jython code,
     9but mysql will not write over an existing file, nor will it write to
     10other random directories.  the easy solution is to create a subdir in
     11/tmp for the output files.
     12
     13This simple modification speeds up the processing of the smfs by a
     14factor of nearly 2x!
     15
     16
    117
    2182013.01.24 : Status of the ippToPsps Upgrade

  • branches/eam_branches/ipp-20121219/ippToPsps/jython/detectionbatch.py

    r35060 r35063  
    927927    def importIppTables(self, filter=""):
    928928
    929        if self.config.test: regex = "XY33.psf"
    930        else : regex = ".*.psf"
     929       ## if self.config.test: regex = "XY33.psf"
     930       ## else : regex = ".*.psf"
     931       regex = ".*.psf"
    931932 
    932        # XXX EAM NOTE : this is fragile : requires PS1_V_
     933       # XXX EAM NOTE : this is fragile : requires PS1_V4
    933934       columns = "IPP_IDET X_PSF Y_PSF X_PSF_SIG Y_PSF_SIG PSF_INST_MAG PSF_INST_MAG_SIG PEAK_FLUX_AS_MAG PSF_MAJOR PSF_MINOR PSF_THETA EXT_NSIGMA PSF_QF MOMENTS_XX MOMENTS_XY MOMENTS_YY AP_MAG KRON_FLUX KRON_FLUX_ERR FLAGS FLAGS2 SKY SKY_SIGMA EXT_NSIGMA PSF_QF_PERFECT PSF_CHISQ"
    934935

  • branches/eam_branches/ipp-20121219/ippToPsps/jython/dvo.py

    r35042 r35063  
    3333
    3434    '''
    35     def __init__(self, logger, config, scratchDbName="ipptopsps_test_scratch"):
     35    def __init__(self, logger, config, scratchDbName=None):
    3636
    3737        # set up logging
    … …  
    4949        self.useStilts = 0
    5050
    51         # connect to the specified scratch database
     51        # connect to the specified scratch database, if a name is given, otherwise use the first version
    5252        try:
    53             self.scratchDb = ScratchDb(self.logger, self.config, scratchDbName)
     53            self.scratchDb = ScratchDb(self.logger, self.config, '1', scratchDbName)
    5454        except: raise
    5555
    … …  
    617617        cmd += " " + str(decCenter+halfSize)
    618618
     619        if self.config.parallel:
     620            cmd += " -parallel"
     621
    619622        self.logger.infoPair("Running dvopsps", cmd)
    620623        p = Popen(cmd, shell=True, stdout=PIPE)

  • branches/eam_branches/ipp-20121219/ippToPsps/jython/dvoobjects.py

    r35056 r35063  
    6767        cmd += " -cpt " + region
    6868
     69        if self.config.parallel:
     70            cmd += " -parallel"
     71
    6972        self.logger.infoPair("Running dvopsps", cmd)
    7073        p = Popen(cmd, shell=True, stdout=PIPE)

  • branches/eam_branches/ipp-20121219/ippToPsps/jython/gpc1db.py

    r35057 r35063  
    2121    '''
    2222    def __init__(self, logger, config):
    23         super(Gpc1Db, self).__init__(logger, config, "gpc1database")
     23        # define database type
     24        if (config.test):
     25            dbType = "gpc1database_test"
     26        else:
     27            dbType = "gpc1database"
     28
     29        super(Gpc1Db, self).__init__(logger, config, dbType)
    2430
    2531    '''

  • branches/eam_branches/ipp-20121219/ippToPsps/jython/ipptopsps.py

    r34970 r35063  
    2222    '''
    2323    def __init__(self, argv, logToStdout=1, logToFile=0):
     24
     25        testmode = False
     26        for arg in sys.argv:
     27            if arg == "-test":
     28                testmode = True
     29                sys.argv.remove(arg)
     30            if arg == "-t":
     31                testmode = True
     32                sys.argv.remove(arg)
    2433
    2534        # are the arsg ok? all programs require a config name
    … …  
    5160        # set up config object
    5261        self.config = Config(self.PROGNAME, CONFIGNAME, self.configDir)
     62        self.config.test = testmode
     63
     64        if self.config.test: print "using test mode"
     65        else: print "not using test mode"
     66
    5367        self.logger = self.config.getLogger(self.HOST, self.PID, logToStdout, logToFile)
    5468

  • branches/eam_branches/ipp-20121219/ippToPsps/jython/ipptopspsdb.py

    r35003 r35063  
    1818    '''
    1919    def __init__(self, logger, config):
    20         super(IppToPspsDb, self).__init__(logger, config, "ipptopspsdatabase")
     20        if (config.test):
     21            dbType = "ipptopspsdatabase_test"
     22        else:
     23            dbType = "ipptopspsdatabase"
     24        super(IppToPspsDb, self).__init__(logger, config, dbType)
    2125
    2226        self.MAX_FAILS = 5
    … …  
    965969        ,queue_ST \
    966970        ,queue_OB \
     971        ,parallel \
    967972        FROM config \
    968973        WHERE name = '" + self.config.name + "'"
    … …  
    10061011            if rs.getInt(20) == 1: self.config.batchTypes.append("ST")
    10071012            if rs.getInt(21) == 1: self.config.batchTypes.append("OB")
     1013
    10081014            self.config.force = True # TODO
    1009             self.config.test = False # TODO
     1015            self.config.parallel = False # TODO
     1016
     1017            if rs.getInt(22) == 1: self.config.parallel = True
     1018
     1019            if self.config.parallel: print "USING parallel"
    10101020            self.config.isLoaded = True
    10111021        except:

  • branches/eam_branches/ipp-20121219/ippToPsps/jython/loader.py

    r35056 r35063  
    3434        super(Loader, self).__init__(argv, 1, 1)
    3535
    36         # create gpc1 database objects
     36        if self.config.parallel:
     37            print "PARALLEL dvo"
     38        else:
     39            print "SERIAL dvo"
     40
     41        # connect to the gpc1 database
    3742        self.gpc1Db = Gpc1Db(self.logger, self.config)
    3843
    39         # XXX this is really bad: need to get these from config... (see this code:)
    40         ## XX # open config and grab database parameters
    41         ## XX if not dbName: self.dbName = config.getDbName(dbType)
    42         ## XX else: self.dbName = dbName
    43 
    4444        # connect to scratch database
    45         scratchDbs = ['ipptopsps_test_scratch', 'ipptopsps_test_scratch2', 'ipptopsps_test_scratch3']
    46         for dbName in scratchDbs:
    47             self.scratchDb = ScratchDb(self.logger, self.config, dbName)
    48             if self.scratchDb.anyOtherConnections():
    49                 self.logger.errorPair("This scratch Db is already in use", dbName)
     45        scratchDbs = ['1', '2', '3']
     46        for dbVersion in scratchDbs:
     47            self.scratchDb = ScratchDb(self.logger, self.config, dbVersion)
     48            if not self.config.test and self.scratchDb.anyOtherConnections():
     49                self.logger.errorPair("This scratch Db is already in use", self.scratchDb.dbName)
    5050                self.scratchDb.disconnect()
    5151                continue

  • branches/eam_branches/ipp-20121219/ippToPsps/jython/scratchdb.py

    r35042 r35063  
    1919    Constructor
    2020    '''
    21     def __init__(self, logger, config, dbName=None):
    22         super(ScratchDb, self).__init__(logger, config, "localdatabase", dbName)
     21    def __init__(self, logger, config, dbVersion, dbName=None):
     22
     23        # create gpc1 database objects
     24        if (config.test):
     25            dbType = "localdatabase_test"
     26        else:
     27            dbType = "localdatabase"
     28
     29        if not dbName:
     30            dbName = config.getDbName(dbType)
     31            dbName += dbVersion
     32
     33        super(ScratchDb, self).__init__(logger, config, dbType, dbName)
    2334
    2435        self.dvoDoneTable = "dvoDone"

  • branches/eam_branches/ipp-20121219/ippToPsps/jython/setupScratchDb.py

    r34990 r35063  
    2626
    2727        try:
    28             self.scratchDb = ScratchDb(self.logger, self.config, argv[2])
     28            self.scratchDb = ScratchDb(self.logger, self.config, '0', argv[2])
    2929        except:
    3030            self.exitProgram("Could not connect to a scratch Db")

  • branches/eam_branches/ipp-20121219/ippToPsps/test/fulltest.sh

    r35051 r35063  
    2525# mkgpc1.sh user localhost eam eam
    2626
    27 set mkgpc1      = 0
    28 set initdb      = 0
    29 set mkdummy     = 0
    30 set initscratch = 0
    31 set initbatch   = 0
    32 set camqueue    = 0
    33 set cambatch    = 0
    34 set stackqueue  = 0
    35 set stackbatch  = 0
     27set mkgpc1       = 1
     28set initdb       = 1
     29set mkdummy      = 1
     30set initscratch  = 1
     31set initbatch    = 1
     32set camqueue     = 1
     33set cambatch     = 1
     34set stackqueue   = 1
     35set stackbatch   = 1
    3636set objectqueue  = 1
    3737set objectbatch  = 1
    … …  
    5656# create a dummy config so setupScratchDb.py does not fail
    5757if ($mkdummy) then
    58   ippjython queue.py edit <<EOF
     58  ippjython queue.py -test edit <<EOF
    5959dummy
    6060none
    … …  
    80800
    81811
     820
    8283EOF
    8384endif
    … …  
    8788
    8889  # create the database (ignore failure if it already exists)
    89   mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps_test_scratch"
     90  mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps_test_scratch1"
    9091  # mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps_scratch2"
    9192  # mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps_scratch3"
    9293
    93   mysql -h localhost -u dvo -pdvo -e "create database ipptopsps_test_scratch"
     94  mysql -h localhost -u dvo -pdvo -e "create database ipptopsps_test_scratch1"
    9495  # mysql -h localhost -u dvo -pdvo -e "create database ipptopsps_scratch2"
    9596  # mysql -h localhost -u dvo -pdvo -e "create database ipptopsps_scratch3"
    9697
    97   ippjython setupScratchDb.py dummy ipptopsps_test_scratch
    98   # ippjython setupScratchDb.py dummy ipptopsps_scratch2
    99   # ippjython setupScratchDb.py dummy ipptopsps_scratch3
     98  ippjython setupScratchDb.py -test dummy ipptopsps_test_scratch1
     99  # ippjython setupScratchDb.py -test dummy ipptopsps_scratch2
     100  # ippjython setupScratchDb.py -test dummy ipptopsps_scratch3
    100101endif
    101102
    102103# this code is sensitive to a previously existing database?
    103104if ($camqueue) then
    104   ippjython queue.py edit <<EOF
     105  ippjython queue.py -test edit <<EOF
    105106$queuename
    106107test_ds
    … …  
    1261270
    1271281
     1290
    128130y
    129131y
    … …  
    133135if ($initbatch) then
    134136  rm -rf $OUTDIR/IN
    135   ippjython loader.py $queuename init
     137  ippjython loader.py -test $queuename init
    136138endif
    137139
    138140if ($cambatch) then
    139141  rm -rf $OUTDIR/P2
    140   ippjython loader.py $queuename once
     142  ippjython loader.py -test $queuename once
    141143endif
    142144
    … …  
    146148    set stackqueuename = $queuename\_stk
    147149  endif
    148   ippjython queue.py edit <<EOF
     150  ippjython queue.py -test edit <<EOF
    149151$stackqueuename
    150152test_ds
    … …  
    1701720
    1711731
     1740
    172175y
    173176y
    … …  
    180183  endif
    181184  rm -rf $OUTDIR/ST
    182   ippjython loader.py $stackqueuename once
     185  ippjython loader.py -test $stackqueuename once
    183186endif
    184187
    … …  
    188191    set objectqueuename = $queuename\_obj
    189192  endif
    190   ippjython queue.py edit <<EOF
     193  ippjython queue.py -test edit <<EOF
    191194$objectqueuename
    192195test_ds
    … …  
    2122151
    2132161
     2170
    214218y
    215219y
    … …  
    222226  endif
    223227  rm -rf $OUTDIR/OB
    224   ippjython loader.py $objectqueuename once
    225 endif
    226 
     228   ippjython loader.py -test $objectqueuename once
     229endif
     230

  • branches/eam_branches/ipp-20121219/ippToPsps/test/mkgpc1data.dvo

    r35051 r35063  
    44
    55if (not($?VERBOSE)) set VERBOSE = 0
     6if (not($?PARALLEL)) set PARALLEL = 0
    67
    78## this script must be run in this directory, output data goes to 'testdata'
    … …  
    199200  end
    200201
     202  if ($PARALLEL)
     203    $hostname = `hostname`
     204    exec rm -f $catdir/HostTable.dat
     205    output $catdir/HostTable.dat
     206    echo "# ID Hostname Catdir"
     207    echo " 1 $hostname $catdir.p1"
     208    echo " 2 $hostname $catdir.p2"
     209    echo " 3 $hostname $catdir.p3"
     210    output stdout
     211    exec dvodist -out $catdir
     212  end
     213
    201214  # for i 0 $offset:n
    202215  #   tapOK {abs(Mcal[$i] - Mcal[0] - $offset:$i) < 0.001} "Mcal $i"
    … …  
    239252
    240253    exec addstar -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass
     254  end
     255
     256  if ($PARALLEL)
     257    $hostname = `hostname`
     258    exec rm -f $catdir/HostTable.dat
     259    output $catdir/HostTable.dat
     260    echo "# ID Hostname Catdir"
     261    echo " 1 $hostname $catdir.p1"
     262    echo " 2 $hostname $catdir.p2"
     263    echo " 3 $hostname $catdir.p3"
     264    output stdout
     265    exec dvodist -out $catdir
    241266  end
    242267
    … …  
    358383
    359384if ($SCRIPT)
     385  if ($argv:n > 0)
     386    if ("$argv:0" == "-parallel")
     387      $PARALLEL = 1
     388    end
     389  end
    360390  mkfull
    361391  exit 0

  • branches/eam_branches/ipp-20121219/ippToPsps/test/sastest.sh

    r35037 r35063  
    1919# mkgpc1.sh user localhost eam eam
    2020
    21 set mkgpc1      = 1
    22 set initdb      = 1
    23 set mkdummy     = 1
    24 set initscratch = 1
    25 set initbatch   = 0
    26 set camqueue    = 0
    27 set cambatch    = 0
     21set mkgpc1      = 0
     22set initdb      = 0
     23set mkdummy     = 0
     24set initscratch = 0
     25set initbatch   = 1
     26set camqueue    = 1
     27set cambatch    = 1
    2828set stackqueue  = 1
    2929set stackbatch  = 1
    … …  
    4141  # does the database already exist?
    4242  # XXX this command will blow away an existing database!!
    43   mysql -h localhost -u dvo -pdvo -e "create database ipptopsps"
    44   mysql -h localhost -u dvo -pdvo ipptopsps < $IPPTOPSPS_DATA/ippToPspsDbSchema.sql
     43  mysql -h localhost -u dvo -pdvo -e "create database ipptopsps_test"
     44  mysql -h localhost -u dvo -pdvo ipptopsps_test < $IPPTOPSPS_DATA/ippToPspsDbSchema.sql
    4545
    4646endif
    … …  
    4848# create a dummy config so setupScratchDb.py does not fail
    4949if ($mkdummy) then
    50   ippjython queue.py edit <<EOF
     50  ippjython queue.py -test edit <<EOF
    5151dummy
    5252none
    … …  
    72720
    73731
     740
     751
    7476EOF
    7577endif
    … …  
    7981
    8082  # create the database (ignore failure if it already exists)
    81   mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps_scratch"
     83  mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps_scratch1"
    8284  # mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps_scratch2"
    8385  # mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps_scratch3"
    8486
    85   mysql -h localhost -u dvo -pdvo -e "create database ipptopsps_scratch"
     87  mysql -h localhost -u dvo -pdvo -e "create database ipptopsps_scratch1"
    8688  # mysql -h localhost -u dvo -pdvo -e "create database ipptopsps_scratch2"
    8789  # mysql -h localhost -u dvo -pdvo -e "create database ipptopsps_scratch3"
    8890
    89   ippjython setupScratchDb.py dummy ipptopsps_scratch
    90   # ippjython setupScratchDb.py dummy ipptopsps_scratch2
    91   # ippjython setupScratchDb.py dummy ipptopsps_scratch3
     91  ippjython setupScratchDb.py -test dummy ipptopsps_scratch1
     92  # ippjython setupScratchDb.py -test dummy ipptopsps_scratch2
     93  # ippjython setupScratchDb.py -test dummy ipptopsps_scratch3
    9294endif
    9395
    9496# this code is sensitive to a previously existing database?
    9597if ($camqueue) then
    96   ippjython queue.py edit <<EOF
     98  ippjython queue.py -test edit <<EOF
    9799$queuename
    98100test_ds
    … …  
    1181200
    1191211
     1220
     1231
    120124y
    121125y
    … …  
    124128
    125129if ($initbatch) then
    126   ippjython loader.py $queuename init
     130  ippjython loader.py -test $queuename init
    127131endif
    128132
    129133if ($cambatch) then
    130   ippjython loader.py $queuename once
     134  ippjython loader.py -test $queuename once
    131135endif
    132136
    … …  
    136140    set stackqueuename = $queuename\_stk
    137141  endif
    138   ippjython queue.py edit <<EOF
     142  ippjython queue.py -test edit <<EOF
    139143$stackqueuename
    140144test_ds
    … …  
    1601640
    1611651
     1660
     1671
    162168y
    163169y
    … …  
    169175    set stackqueuename = $queuename\_stk
    170176  endif
    171   ippjython loader.py $stackqueuename once
     177  ippjython loader.py -test $stackqueuename once
    172178endif
    173179
Note: See TracChangeset for help on using the changeset viewer.