IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 35554


Ignore:
Timestamp:
May 9, 2013, 11:35:03 AM (13 years ago)
Author:
eugene
Message:

merge changes from trunk

Location:
branches/eam_branches/ipp-20130419
Files:
19 edited

Legend:

Unmodified
Added
Removed

  • branches/eam_branches/ipp-20130419

  • branches/eam_branches/ipp-20130419/DataStore/lib/DataStore/Utils.pm

    r35411 r35554  
    4444%KNOWN_FILE_TYPES = map { $_ => 1 } qw( chip psrequest psresults pstamp chipproc warp stack diff ipp-mops table text xml tgz fits IPP-MOPS IPP-PSPS ipp-psps notset );
    4545%KNOWN_FILESET_TYPES = map { $_ => 1 } qw( OBJECT BIAS DARK SKYFLAT DOMEFLAT OOF SHACKHARTMANN PSREQUEST PSRESULTS IPP-MOPS XRAY FOCUS MOPS_DETECTABILITY_QUERY MOPS_DETECTABILITY_RESPONSE MOPS_TRANSIENT_DETECTIONS LED notset IPP_PSPS IPP-DIST table);
    46 %KNOWN_PRODUCT_TYPES = map { $_ => 1 } qw( image dump psrequest psresults table ipp-dist ipp-misc dqresults);
     46%KNOWN_PRODUCT_TYPES = map { $_ => 1 } qw( image dump psrequest psresults table ipp-dist ipp-misc dqresults IPP-MOPS IPP-PSPS PSRESULTS);
    4747
    4848=pod

  • branches/eam_branches/ipp-20130419/Nebulous-Server

    • Property svn:mergeinfo changed (with no actual effect on merging)

  • branches/eam_branches/ipp-20130419/doc/stamps/stampsICD.tex

    r33884 r35554  
    99\project{Pan-STARRS IPP}
    1010\organization{Institute for Astronomy}
    11 \version{1.0}
     11\version{2.0}
    1212\docnumber{PSDC-940-011-00}
    1313
    … …  
    4848one or more FITS binary tables called \emph{request tables}.
    4949Each row in a request table contains a postage stamp request specification.
    50 Each request specification causes zero or more postage stamp jobs to be processed.
     50Each request specification causes zero or more postage stamp jobs to be created.
    5151Images are added to the output file set for each successful job.
    5252
    5353The request specification give the parameters that the PSS uses to determine the images of interest
    5454and the Region of Interest.
     55
     56Note: The value "null" for strings or 0 for numbers may be used to disable the application of
     57certain parameters.
    5558
    5659\subsection{Request Table Contents}
    … …  
    6972\hline
    7073\code{EXTNAME}  & string     & \code{PS1_PS_REQUEST} \\
    71 \code{EXTVER}   & string     & 1  \\
     74\code{EXTVER}   & string     & 2  \\
    7275\code{REQ_NAME} & string     & unique name of the request (see text) \\
     76\code{ACTION}   & string     & PROCESS \\
     77\code{EMAIL}    & string     & Email address of submitting user\\
    7378\hline
    7479\end{tabular}
    … …  
    8186This value is used for the name of the request's output file set.
    8287
    83 The postage stamp server supports three types of "postage stamp" job specified by the value
     88
     89The postage stamp server supports two types of "postage stamp" job specified by the value
    8490the column \code{JOB_TYPE}. If the value is \emph{stamp} postage stamp images will be extracted
    8591from the images of interest.  If the value of \code{JOB_TYPE} is \emph{get\_image}
    86 the data from the selected images' "runs"  with (satellite streaks removed) are bundled
    87 in and placed on the output file set.
     92the data from the selected images' "runs" are bundled in the format used by the IPP distribution system
     93and are placed on the output file set.
    8894
    8995\begin{table}[h]
    … …  
    97103\code{ROWNUM}   & U32       &                           &        must be unique in file\\
    98104\code{PROJECT}  & string    & gpc1                      &        \\
     105\code{SURVEY_NAME}  & string    & PS1 survey names      & see table 3        \\
     106\code{IPP_RELEASE}  & string    & IPP release name      & e.g. 3PI.PV1, 3PI.nightly \\
    99107\code{JOB_TYPE} & string    & stamp or get\_image       &        \\
    100108\\
    … …  
    117125\code{MJD_MIN}  &   F64     & Modified Julian Day       & \\
    118126\code{MJD_MAX}  &   F64     &                           & \\
    119 \\
    120 \code{COMMENT}  &   string  & copied to the results flie & \\
     127\code{RUN_TYPE}  &  string  & stack or diff type        & deep, reference, nightly, warp-stack, etc.\\
     128\code{FWHM_MIN}  &   F64    & minimum measured seeing (pixels) & \\
     129\code{FWHM_MAX}  &   F64    & maximum measured seeing (pixels) & \\
     130\\
     131\code{COMMENT}  &   string  & copied to the results file & \\
    121132\hline
    122133\end{tabular}
    … …  
    159170\end{itemize}
    160171
    161 Note that at this time the REQ\_TYPE byexp is not implemented for IMG\_TYPE stack.
     172Note that REQ\_TYPE byexp is not implemented for IMG\_TYPE stack.
    162173
    163174COMPONENT must also be supplied for request specifications where all of the following are true
    … …  
    173184\end{itemize}
    174185
    175 The special value 'all' may be used to select all component images but this is not recommended.
     186The special value "all" may be used to select all component components (chips or skycells) of a run.
     187
    176188
    177189\subsubsection{Image Selection Cuts}\label{ImageCuts}
    178190
    179 A request specification may refer to many images. The scope of the search can be
    180 limited to certain dates with the parameters MJD\_MIN and MJD\_MAX. The specified range is inclusive.
    181 Note: these cuts have no effect for stack stage requests.
     191A request specification may refer to many images. The scope of the search can be reduced by providing
     192the selection parameters.
     193
     194SURVEY\_NAME (new in version 2) restricts the queries to images for a given PS1 survey. Valid values are
     195shown in the table below. The value "null" may also be used.
     196
     197\begin{table}[h]
     198\begin{center}
     199\caption{PS1 SURVEYNAME Values}\label{surveys}
     200\begin{tabular}{ll}
     201\hline
     202\hline
     203{\bf SURVEY\_NAME} &  {\bf Description } \\
     204\hline
     2053PI  & PS1 3PI Survey \\
     206MD01 & PS1 MD01 XMM-LSS-DXS 022224-043000 \\
     207MD02 & PS1 MD02 CDFS/GOODS/GEMS 033224-274800 \\
     208MD03 & PS1 MD03 IFA/Lynx 084300+444000 \\
     209MD04 & PS1 MD04 COSMOS 100000+021200 \\
     210MD05 & PS1 MD05 Lockman-DXS 110000+570000 \\
     211MD06 & PS1 MD06 NGC 4258 121857+471814 \\
     212MD07 & PS1 MD07 VISTA-Video1 140000+050000 \\
     213MD08 & PS1 MD08 EliasN1-DXS 160000+570000 \\
     214MD09 & PS1 MD09 Vimos4-DXS-SSA 220000+003000 \\
     215MD10 & PS1 MD10 Deep2 233000+000000 \\
     216M31  & PS1 M31 Andromeda 004242+411600 \\
     217STS1 & PS1 STS1 Outskirts of Bulge 195048+170339 \\
     218STS2 & PS1 STS2 Hyades 004000+150000 \\
     219STS3 & PS1 STS3 Praesepe 083000+200000 \\
     220SSS  & PS1 Solar System Survey \\
     221\hline
     222\end{tabular}
     223\end{center}
     224\end{table}
     225
     226IPP\_RELEASE (new in version 2) restricts the search for data from a particular "release" of IPP data.
     227If "null" the system choses data from the highest priority release ("best") of the given exposure or skycell.
    182228
    183229DATA\_GROUP restricts the search to runs with the given value. Note that the SQL database lookups
    184 are done with a "LIKE" comparison so SQL wild cards may be used.
     230are done with a "LIKE" comparison so SQL wild cards may be used. With Version 2 of this specification
     231use of DATA\_GROUP is deprecated. It is ignored if a non null value for SURVEY\_NAME or IPP\_RELEASE is
     232supplied.
     233
     234Searches may be limited to certain dates with the parameters MJD\_MIN and MJD\_MAX. The specified range is inclusive.
     235Note: For stack stage requests, these cuts will have no effect on queries for deep or reference stacks.
    185236
    186237The REQFILT parameter may be used to restrict the search for images taken with a given filter.
    187 If the filter is a single character (g, r, i, z, y) the server appends a '\%' character to the
    188 sql query.
     238If the filter is a single character (g, r, i, z, y, w) the server appends a '\%' character to the
     239sql query. If "null" data from all filters is queried.
     240
     241The optional parameter RUN\_TYPE is used for specify the type of stack or difference images to consider as
     242images of interest.
     243
     244For difference images valid values are warp\_warp warp\_stack and stack\_stack.
     245
     246For stacks valid values are "deep", "reference", "nightly", and "notnightly". For stacks, this selector
     247is only applied if SURVEY\_NAME or IPP\_RELEASE is supplied.
     248
    189249
    190250
    … …  
    230290When pixel coordinates are used the coordinate system is that of the image. (For chip level images, before
    231291the stamps are created the cells are mosaicked into a single image.)
     292
     293Note: If the width and height are both equal to zero, the entire input image is included in the stamp. Otherwise
     294values less than or equal to zero will generate an error.
    232295
    233296\subsection{Request Options}
    … …  
    252315PSTAMP\_SELECT\_INVERSE   &    1024  &   make stamps from inverse difference images (applies only for diff) \\
    253316PSTAMP\_SELECT\_UNCONV    &    2048  &   make stamps from unconvolved stack images (applies only for stack) \\
    254 PSTAMP\_RESTORE\_BACKGROUND &  4096  &   (TODO) restore background correction to chip stage images before making stamp \\
     317PSTAMP\_RESTORE\_BACKGROUND &  4096  &   restore background correction to chip stage images before making stamp \\
    255318PSTAMP\_USE\_IMFILE\_ID   &   16384  &   for byid requests select images using image id instead of stage id \\
    256319PSTAMP\_NO\_WAIT\_FOR\_UPDATE &  32768  &   if images are not available because they have been cleaned don't wait \\
    … …  
    299362the pixels for requested images, if possible. This is referred to as the update process.
    300363If the user does not wish to wait for the updates to complete he should include
    301 \code{PSTAMP_NO_WAIT_FOR_UPDATE} in the \code{OPTION_MASK}.
     364\code{PSTAMP_NO_WAIT_FOR_UPDATE} in the \code{OPTION_MASK}. In this case the row will receive the error
     365code PSTAMP\_NOT\_AVAIALABLE.
    302366
    303367
    … …  
    353417\code{EXPTIME}      & F64       &  exposure time of parent image \\
    354418\code{FPA_ID}       & string    &  orginal FPA\_ID used at injection \\
    355 \\
    356 \code{PROJECT}  & string    & Values from initial request specification\\
     419\hline
     420& Values from corresponding row in the request file\\
     421\hline
     422\code{PROJECT}  & string    & \\
     423\code{SURVEY_NAME}  & string    & \\
     424\code{IPP_RELEASE}  & string    & \\
    357425\code{JOB_TYPE} & string    & \\
    358426\\
    … …  
    372440\code{MJD_MAX}  &   F64     & \\
    373441\code{DATA_GROUP}  &   string     & \\
     442\code{RUN_TYPE}  &   string     & \\
     443\code{FWHM_MIN}  &   F64     & \\
     444\code{FWHM_MAX}  &   F64     & \\
    374445\code{COMMENT}  &   string     & \\
    375446\\
    … …  
    393464have the name \code{ROWNUM_I.mk.fits}
    394465
    395 JOB\_ID is the value of the postage stamp server's internal job id. (This is provided primarily as a debugging
     466JOB\_ID is the value of the postage stamp server's internal job id.
    396467aid.)
    397468
    398 The column named PROJECT and following columns are copies from the input request specification.
     469The column named PROJECT and following columns are copied from the input request specification.
    399470
    400471If ERROR\_CODE is non-zero an error occurred when processing the request specification or the job.
    … …  
    419490PSTAMP\_UNKNOWN\_PRODUCT  & 22 &    Unknown product in request specification \\
    420491PSTAMP\_NO\_IMAGE\_MATCH  & 23 &    No images matched the request \\
    421 PSTAMP\_NOT\_DESTREAKED   & 24 &    Image matched, but not yet de-streaked \\
     492PSTAMP\_NOT\_DESTREAKED   & 24 &    Obsolete - No longer used\\
    422493PSTAMP\_NOT\_AVAILABLE    & 25 &    Image not available (temporary) \\
    423494PSTAMP\_GONE              & 26 &    Image is no longer available \\
    424495PSTAMP\_NO\_JOBS\_QUEUED  & 27 &    Request specification yielded no jobs. See parse\_error.txt \\
    425496PSTAMP\_NO\_OVERLAP       & 28 &    Center coordinates are not contained in any image of interest \\
     497PSTAMP\_NOT\_AUTHORIZED   & 29 &    user is not allowed access to the selected data\\
    426498
    427499\hline

  • branches/eam_branches/ipp-20130419/ippToPsps

  • branches/eam_branches/ipp-20130419/ippToPsps/config/ippToPspsDbSchema.sql

    r35232 r35554  
    4848  `purged` tinyint(4) NOT NULL default '0',
    4949  PRIMARY KEY  (`batch_id`)
    50 ) ENGINE=InnoDB AUTO_INCREMENT=328286 DEFAULT CHARSET=latin1;
     50) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1;
    5151SET character_set_client = @saved_cs_client;
    5252
    … …  
    6969  KEY `fk_skychunk` (`skychunk`),
    7070  CONSTRAINT `fk_skychunk` FOREIGN KEY (`skychunk`) REFERENCES `skychunk` (`name`) ON DELETE CASCADE
    71 ) ENGINE=InnoDB AUTO_INCREMENT=44588 DEFAULT CHARSET=latin1;
     71) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1;
    7272SET character_set_client = @saved_cs_client;
    7373
    … …  
    9191  PRIMARY KEY  (`id`),
    9292  UNIQUE KEY `unique_key` (`host`,`pid`)
    93 ) ENGINE=InnoDB AUTO_INCREMENT=1227 DEFAULT CHARSET=latin1;
     93) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1;
    9494SET character_set_client = @saved_cs_client;
    9595

  • branches/eam_branches/ipp-20130419/ippToPsps/config/settings.xml

    r35417 r35554  
    4545      <memory>16</memory>
    4646    </loader>
     47    <loader mach="ipp045">
     48      <name>ipptopsps_scratch</name>
     49      <host>ipp045</host>
     50      <user>dvo</user>
     51      <password>dvo</password>
     52      <memory>32</memory>
     53    </loader>
     54    <loader mach="ipp046">
     55      <name>ipptopsps_scratch</name>
     56      <host>ipp046</host>
     57      <user>dvo</user>
     58      <password>dvo</password>
     59      <memory>32</memory>
     60    </loader>
     61    <loader mach="ipp047">
     62      <name>ipptopsps_scratch</name>
     63      <host>ipp047</host>
     64      <user>dvo</user>
     65      <password>dvo</password>
     66      <memory>32</memory>
     67    </loader>
     68    <loader mach="ipp048">
     69      <name>ipptopsps_scratch</name>
     70      <host>ipp048</host>
     71      <user>dvo</user>
     72      <password>dvo</password>
     73      <memory>32</memory>
     74    </loader>
     75    <loader mach="ipp049">
     76      <name>ipptopsps_scratch</name>
     77      <host>ipp049</host>
     78      <user>dvo</user>
     79      <password>dvo</password>
     80      <memory>32</memory>
     81    </loader>
     82    <loader mach="ipp050">
     83      <name>ipptopsps_scratch</name>
     84      <host>ipp050</host>
     85      <user>dvo</user>
     86      <password>dvo</password>
     87      <memory>32</memory>
     88    </loader>
     89    <loader mach="ipp051">
     90      <name>ipptopsps_scratch</name>
     91      <host>ipp051</host>
     92      <user>dvo</user>
     93      <password>dvo</password>
     94      <memory>32</memory>
     95    </loader>
     96    <loader mach="ipp052">
     97      <name>ipptopsps_scratch</name>
     98      <host>ipp052</host>
     99      <user>dvo</user>
     100      <password>dvo</password>
     101      <memory>32</memory>
     102    </loader>
     103    <loader mach="ipp053">
     104      <name>ipptopsps_scratch</name>
     105      <host>ipp053</host>
     106      <user>dvo</user>
     107      <password>dvo</password>
     108      <memory>32</memory>
     109    </loader>
     110    <loader mach="ipp054">
     111      <name>ipptopsps_scratch</name>
     112      <host>ipp054</host>
     113      <user>dvo</user>
     114      <password>dvo</password>
     115      <memory>32</memory>
     116    </loader>
     117    <loader mach="ipp055">
     118      <name>ipptopsps_scratch</name>
     119      <host>ipp055</host>
     120      <user>dvo</user>
     121      <password>dvo</password>
     122      <memory>32</memory>
     123    </loader>
     124    <loader mach="ipp056">
     125      <name>ipptopsps_scratch</name>
     126      <host>ipp056</host>
     127      <user>dvo</user>
     128      <password>dvo</password>
     129      <memory>32</memory>
     130    </loader>
     131    <loader mach="ipp057">
     132      <name>ipptopsps_scratch</name>
     133      <host>ipp057</host>
     134      <user>dvo</user>
     135      <password>dvo</password>
     136      <memory>32</memory>
     137    </loader>
     138    <loader mach="ipp058">
     139      <name>ipptopsps_scratch</name>
     140      <host>ipp058</host>
     141      <user>dvo</user>
     142      <password>dvo</password>
     143      <memory>32</memory>
     144    </loader>
     145    <loader mach="ipp059">
     146      <name>ipptopsps_scratch</name>
     147      <host>ipp059</host>
     148      <user>dvo</user>
     149      <password>dvo</password>
     150      <memory>32</memory>
     151    </loader>
     152    <loader mach="ipp060">
     153      <name>ipptopsps_scratch</name>
     154      <host>ipp060</host>
     155      <user>dvo</user>
     156      <password>dvo</password>
     157      <memory>32</memory>
     158    </loader>
     159    <loader mach="ipp061">
     160      <name>ipptopsps_scratch</name>
     161      <host>ipp061</host>
     162      <user>dvo</user>
     163      <password>dvo</password>
     164      <memory>32</memory>
     165    </loader>
     166    <loader mach="ipp062">
     167      <name>ipptopsps_scratch</name>
     168      <host>ipp062</host>
     169      <user>dvo</user>
     170      <password>dvo</password>
     171      <memory>32</memory>
     172    </loader>
     173    <loader mach="ipp063">
     174      <name>ipptopsps_scratch</name>
     175      <host>ipp063</host>
     176      <user>dvo</user>
     177      <password>dvo</password>
     178      <memory>32</memory>
     179    </loader>
    47180    <loader mach="ipp064">
    48181      <name>ipptopsps_scratch</name>
    … …  
    52185      <memory>32</memory>
    53186    </loader>
     187    <loader mach="ipp065">
     188      <name>ipptopsps_scratch</name>
     189      <host>ipp065</host>
     190      <user>dvo</user>
     191      <password>dvo</password>
     192      <memory>32</memory>
     193    </loader>
     194    <loader mach="ipp066">
     195      <name>ipptopsps_scratch</name>
     196      <host>ipp066</host>
     197      <user>dvo</user>
     198      <password>dvo</password>
     199      <memory>32</memory>
     200    </loader>
     201    <loader mach="stare00">
     202      <name>ipptopsps_scratch</name>
     203      <host>stare00</host>
     204      <user>dvo</user>
     205      <password>dvo</password>
     206      <memory>64</memory>
     207    </loader>
     208    <loader mach="stare01">
     209      <name>ipptopsps_scratch</name>
     210      <host>stare01</host>
     211      <user>dvo</user>
     212      <password>dvo</password>
     213      <memory>64</memory>
     214    </loader>
     215    <loader mach="stare02">
     216      <name>ipptopsps_scratch</name>
     217      <host>stare02</host>
     218      <user>dvo</user>
     219      <password>dvo</password>
     220      <memory>64</memory>
     221    </loader>
     222    <loader mach="stare03">
     223      <name>ipptopsps_scratch</name>
     224      <host>stare03</host>
     225      <user>dvo</user>
     226      <password>dvo</password>
     227      <memory>64</memory>
     228    </loader>
     229    <loader mach="stare04">
     230      <name>ipptopsps_scratch</name>
     231      <host>stare04</host>
     232      <user>dvo</user>
     233      <password>dvo</password>
     234      <memory>64</memory>
     235    </loader>
     236
     237
     238
     239
     240
    54241    <loader mach="generic">
    55242      <name>ipptopsps_scratch</name>

  • branches/eam_branches/ipp-20130419/ippToPsps/doc/notes.txt

    r35233 r35554  
     1
     22013.04.27
     3
     4 need to move the conversion between mags and flux into dvopsps.
     5
     6 * detection table
     7   * psfFlux : uses PSF_INST_MAG instead of PSF_INST_FLUX
     8   * psfFluxErr : uses PSF_INST_MAG_SIG and PSF_INST_MAG instead of PSF_INST_FLUX_SIG
     9   * apFlux : uses AP_MAG instead of AP_FLUX
     10   * kronFlux : correctly uses KRON_FLUX
     11   * kronFluxErr : correctly uses KRON_FLUX_ERR
     12   
     13   ** psfFlux is forced to 0.0000001 instead of 0.0 -- why?
     14   
     15 * stack detection table (same issues as detection table)
     16   * psfFlux : uses PSF_INST_MAG instead of PSF_INST_FLUX
     17   * psfFluxErr : uses PSF_INST_MAG_SIG and PSF_INST_MAG instead of PSF_INST_FLUX_SIG
     18   * apFlux : uses AP_MAG instead of AP_FLUX
     19   * kronFlux : correctly uses KRON_FLUX
     20   * kronFluxErr : correctly uses KRON_FLUX_ERR
     21   
     22   * psfFlux is set to 2 sigma if psfFlux is NULL
     23
     24 * object
     25   * Mean PSF Mag : direct from DVO MAG
     26   * Mean PSF Mag Err : direct from DVO MAG_ERR
     27   * Mean Kron Mag : direct from DVO KRON MAG
     28   * Mean Kron Mag Err : direct from DVO KRON MAG_ERR
     29   
     30   * Stack PSF Mag : from DVO stack psf flux
     31   * Stack PSF Mag Err : from DVO stack psf flux err
     32   * Stack Kron Mag : from DVO stack kron flux
     33   * Stack Kron Mag Err : from DVO stack kron flux err
     34     
     35 
     36
     37   
     38
     39old notes:
    140
    241example of joining ipptopsps and gpc1 tables:

  • branches/eam_branches/ipp-20130419/ippToPsps/jython/batch.py

    r35417 r35554  
    6565        self.scratchDb = scratchDb
    6666        self.batchType = batchType;
     67        self.skipBatch = False; # set to true if there is a non-fatal error
    6768        self.pspsVoTableFilePath = self.config.configDir + "tables." + batchType + ".vot"
    6869

  • branches/eam_branches/ipp-20130419/ippToPsps/jython/detectionbatch.py

    r35417 r35554  
    439439               ,X_PSF_SIG \
    440440               ,Y_PSF_SIG \
    441                ,POW(10.0, (-0.4*PSF_INST_MAG)) / " + self.header['EXPTIME'] + " \
    442                ,ABS((PSF_INST_MAG_SIG*(POW(10.0, (-0.4*PSF_INST_MAG)) / " + self.header['EXPTIME'] + ")) / 1.085736) \
     441               ,PSF_INST_FLUX / " + self.header['EXPTIME'] + " \
     442               ,PSF_INST_FLUX_SIG / " + self.header['EXPTIME'] + " \
    443443               ,POW(10.0, (-0.4*PEAK_FLUX_AS_MAG)) / " + self.header['EXPTIME'] + " \
    444444               ,PSF_MAJOR \
    … …  
    450450               ,MOMENTS_XY \
    451451               ,MOMENTS_YY \
    452                ,POW(10.0, -0.4 *AP_MAG) / " + self.header['EXPTIME'] + " \
     452               ,AP_FLUX / " + self.header['EXPTIME'] + " \
    453453               ,KRON_FLUX / " + self.header['EXPTIME'] + " \
    454454               ,KRON_FLUX_ERR / " + self.header['EXPTIME'] + " \
    … …  
    466466        # self.logger.info(sql)
    467467
    468         self.scratchDb.execute(sql)
    469 
     468# these were used above (left over from before PSF_INST_FLUX, PSF_INST_FLUX_SIG were available?
     469#               ,POW(10.0, (-0.4*PSF_INST_MAG)) / " + self.header['EXPTIME'] + "
     470#               ,ABS((PSF_INST_MAG_SIG*(POW(10.0, (-0.4*PSF_INST_MAG)) / " + self.header['EXPTIME'] + ")) / 1.085736)
     471#               ,POW(10.0, -0.4 *AP_MAG) / " + self.header['EXPTIME'] + "
     472
     473        self.scratchDb.execute(sql)
     474
     475        # XXX EAM : I removed this old fix for invalid fluxes
    470476        # add a instFlux = 0.0 -> 0.000001
    471         # XXX EAM : why is this done?
    472         sql="UPDATE " + pspsTableName + " SET psfFlux = 0.0000001 WHERE psfFlux =  0"
    473         self.scratchDb.execute(sql)
     477        # XXX EAM : why is this done? 
     478        # sql="UPDATE " + pspsTableName + " SET psfFlux = 0.0000001 WHERE psfFlux =  0"
     479        # self.scratchDb.execute(sql)
    474480     
     481        # NOTE : Flux limits : in the current PSPS schema, negative fluxes
     482        # cause problems for sql queries which work in mags as SQL cannot do
     483        # something like (f < 0.0) ? -999 : -2.5*log10(f)
     484        # as a result, the negative fluxes here result in floating point errors
     485        sql = "UPDATE " + pspsTableName + " SET psfFlux = 1e20 WHERE psfFlux <= 0.0"
     486        self.scratchDb.execute(sql)
     487       
     488        sql = "UPDATE " + pspsTableName + " SET apFlux = 1e20 WHERE apFlux <= 0.0"
     489        self.scratchDb.execute(sql)
     490
     491        sql = "UPDATE " + pspsTableName + " SET kronFlux = 1e20 WHERE kronFlux <= 0.0"
     492        self.scratchDb.execute(sql)
     493
    475494        # we don't delete these anymore
    476495        results['NULLINSTFLUX'] = 0;
    … …  
    10161035 
    10171036       # XXX EAM NOTE : this is fragile : requires PS1_V4
    1018        columns = "IPP_IDET X_PSF Y_PSF X_PSF_SIG Y_PSF_SIG PSF_INST_MAG PSF_INST_MAG_SIG PEAK_FLUX_AS_MAG PSF_MAJOR PSF_MINOR PSF_THETA EXT_NSIGMA PSF_QF MOMENTS_XX MOMENTS_XY MOMENTS_YY AP_MAG KRON_FLUX KRON_FLUX_ERR FLAGS FLAGS2 SKY SKY_SIGMA EXT_NSIGMA PSF_QF_PERFECT PSF_CHISQ"
     1037       columns = "IPP_IDET X_PSF Y_PSF X_PSF_SIG Y_PSF_SIG PSF_INST_MAG PSF_INST_MAG_SIG PSF_INST_FLUX PSF_INST_FLUX_SIG PEAK_FLUX_AS_MAG PSF_MAJOR PSF_MINOR PSF_THETA EXT_NSIGMA PSF_QF MOMENTS_XX MOMENTS_XY MOMENTS_YY AP_MAG AP_FLUX KRON_FLUX KRON_FLUX_ERR FLAGS FLAGS2 SKY SKY_SIGMA EXT_NSIGMA PSF_QF_PERFECT PSF_CHISQ"
    10191038
    10201039       return super(DetectionBatch, self).importIppTables(columns, regex)

  • branches/eam_branches/ipp-20130419/ippToPsps/jython/objectbatch.py

    r35417 r35554  
    137137                   ," + filter[1] + "MeanKronMag = MAG_KRON \
    138138                   ," + filter[1] + "MeanKronMagErr = MAG_KRON_ERR \
    139                    ," + filter[1] + "StackPSFMag = -2.5 * log10(FLUX_PSF) + 8.9 \
    140                    ," + filter[1] + "StackPSFMagErr = FLUX_PSF_ERR / FLUX_PSF \
    141                    ," + filter[1] + "StackKronMag = -2.5 * log10(FLUX_KRON) + 8.9 \
    142                    ," + filter[1] + "StackKronMagErr = FLUX_KRON_ERR / FLUX_KRON \
     139                   ," + filter[1] + "StackPSFMag = STACK_PSF_MAG \
     140                   ," + filter[1] + "StackPSFMagErr = STACK_PSF_MAG_ERR \
     141                   ," + filter[1] + "StackKronMag = STACK_KRON_MAG \
     142                   ," + filter[1] + "StackKronMagErr = STACK_KRON_MAG_ERR \
    143143                   ," + filter[1] + "20pct = MAG_20/1000 \
    144144                   ," + filter[1] + "80pct = MAG_80/1000 \
    145145                   ," + filter[1] + "Flags = (0x7fff & FLAGS) | ((FLAGS >> 11) & 0x2000) \
    146146                   ," + filter[1] + "StackDetectID = STACK_DETECT_ID"
     147
     148# old method for setting the Stack (PSF,Kron) Mags:
     149#                  ," + filter[1] + "StackPSFMag = -2.5 * log10(FLUX_PSF) + 8.9 \
     150#                  ," + filter[1] + "StackPSFMagErr = FLUX_PSF_ERR / FLUX_PSF \
     151#                  ," + filter[1] + "StackKronMag = -2.5 * log10(FLUX_KRON) + 8.9 \
     152#                  ," + filter[1] + "StackKronMagErr = FLUX_KRON_ERR / FLUX_KRON \
    147153
    148154            #self.logger.info(sql)
    … …  
    216222        cpsTableName = self.scratchDb.getDbFriendlyTableName(self.region + ".cps")
    217223
    218         self.logger.infoPair("setting to null  > 1e-38 and < 1e-38 in", "cps FLUX_KRON_ERR")
    219         sql = "UPDATE " + cpsTableName + " set FLUX_KRON_ERR = NULL where FLUX_KRON_ERR < 1e-37 AND FLUX_KRON_ERR > -1e-37 "
     224        if False:
     225            self.logger.infoPair("setting to null  > 1e-38 and < 1e-38 in", "cps FLUX_KRON_ERR")
     226            sql = "UPDATE " + cpsTableName + " set FLUX_KRON_ERR = NULL where FLUX_KRON_ERR < 1e-37 AND FLUX_KRON_ERR > -1e-37 "
    220227     
    221         try:
    222             self.scratchDb.execute(sql)
    223         except:
    224             self.logger.errorPair("Couldn't squash out of range stuff", sql)
    225             return False
    226 
    227         self.logger.infoPair("setting to null  > 1e-38 and < 1e-38 in", "cps FLUX_KRON")
    228         sql = "UPDATE " + cpsTableName + " set FLUX_KRON = NULL where FLUX_KRON < 1e-37 AND FLUX_KRON > -1e-37 "
     228            try:
     229                self.scratchDb.execute(sql)
     230            except:
     231                self.logger.errorPair("Couldn't squash out of range stuff", sql)
     232                return False
     233
     234            self.logger.infoPair("setting to null  > 1e-38 and < 1e-38 in", "cps FLUX_KRON")
     235            sql = "UPDATE " + cpsTableName + " set FLUX_KRON = NULL where FLUX_KRON < 1e-37 AND FLUX_KRON > -1e-37 "
    229236     
    230         try:
    231             self.scratchDb.execute(sql)
    232         except:
    233             self.logger.errorPair("Couldn't squash out of range stuff", sql)
    234             return False
    235 
    236 
    237 
     237            try:
     238                self.scratchDb.execute(sql)
     239            except:
     240                self.logger.errorPair("Couldn't squash out of range stuff", sql)
     241                return False
    238242
    239243       

  • branches/eam_branches/ipp-20130419/ippToPsps/jython/stackbatch.py

    r35417 r35554  
    482482               ,X_PSF_SIG \
    483483               ,Y_PSF_SIG \
    484                ,POW(10.0, (-0.4*PSF_INST_MAG)) / " + str(self.expTime) + " \
    485                ,ABS((PSF_INST_MAG_SIG*(POW(10.0, (-0.4*PSF_INST_MAG)) / " + str(self.expTime) + ")) / 1.085736) \
     484               ,PSF_INST_FLUX / " + str(self.expTime) + " \
     485               ,PSF_INST_FLUX_SIG / " + str(self.expTime) + " \
    486486               ,POW(10.0, (-0.4*PEAK_FLUX_AS_MAG)) / " + str(self.expTime) + " \
    487487               ,SKY / " + str(self.expTime) + " \
    … …  
    504504               ,MOMENTS_R1 \
    505505               ,MOMENTS_RH \
    506                ,POW(10.0, -0.4*AP_MAG) / " + str(self.expTime) + " \
     506               ,AP_FLUX / " + str(self.expTime) + " \
    507507               ,NULL \
    508508               ,KRON_FLUX / " + str(self.expTime) + " \
    … …  
    515515               FROM SkyChip_psf"
    516516
     517# these were used above (left over from before PSF_INST_FLUX, PSF_INST_FLUX_SIG were available?
     518#               ,POW(10.0, (-0.4*PSF_INST_MAG)) / " + str(self.expTime) + "
     519#               ,ABS((PSF_INST_MAG_SIG*(POW(10.0, (-0.4*PSF_INST_MAG)) / " + str(self.expTime) + ")) / 1.085736)
     520#               ,POW(10.0, -0.4*AP_MAG) / " + str(self.expTime) + "
     521
    517522        # print "sql: ", sql
    518523        # response = raw_input("ready to insert stack det ")
    … …  
    526531        self.logger.infoPair("Deleting", "entries with StackDetection.objID = 0")
    527532       
     533        # XXX EAM : this seems quite inefficient : these commands use updates to set
     534        # fields which are constant across all rows.  Why not just include these in
     535        # the original insert above?
     536
    528537        self.scratchDb.updateAllRows("StackDetection", "surveyID", str(self.surveyID))
    529538       
    … …  
    560569        # response = raw_input("add primary key? ")
    561570 
    562         if self.stackType == "DEEP_STACK":
     571        ### XXX this should not be a DEEP_STACK only case...
     572        if False and self.stackType == "DEEP_STACK":
    563573
    564574            #if deep stack and instFlux = null and err not null
    … …  
    572582            # response = raw_input("add psf flux ")
    573583           
     584        # NOTE : Flux limits : in the current PSPS schema, negative fluxes
     585        # cause problems for sql queries which work in mags as SQL cannot do
     586        # something like (f < 0.0) ? -999 : -2.5*log10(f)
     587        # as a result, the negative fluxes here result in floating point errors
     588        sql = "UPDATE StackDetection SET psfFlux = 1e20 WHERE psfFlux <= 0.0"
     589        self.scratchDb.execute(sql)
     590       
     591        sql = "UPDATE StackDetection SET apFlux = 1e20 WHERE apFlux <= 0.0"
     592        self.scratchDb.execute(sql)
     593
     594        sql = "UPDATE StackDetection SET kronFlux = 1e20 WHERE kronFlux <= 0.0"
     595        self.scratchDb.execute(sql)
     596
    574597        #leave null instflux in
    575598        #self.scratchDb.reportAndDeleteRowsWithNULLS("StackDetection", "instFlux")
    … …  
    930953    def populatePspsTables(self):
    931954
     955        self.skipBatch = False
     956
    932957        if not self.useFullTables:
    933958            if not self.getIDsFromDVO():

  • branches/eam_branches/ipp-20130419/ippToPsps/test/fulltest.sh

    r35417 r35554  
    11#!/bin/csh -f
     2
     3chmod +x *.dvo *.sh
    24
    35# a bit crude : get the datadir path by interpollation of the PATH element for psconfig
    … …  
    2325set objectbatch  = 1
    2426
    25 set args    = ""
    26 
    27 while ("$1" != "")
    28  switch ("$1")
     27set targ = ($argv)
     28set args = ""
     29
     30while ($#targ)
     31 switch ("$targ[1]")
     32  case -h
     33  case -help
     34    echo "USAGE: fulltest.sh [-XX] [+XX]"
     35    echo " -XX : turn off features"
     36    echo " +XX : turn on features"
     37    echo "  features: all, mkgpc1, initdb, initscratch, initbatch, camera, stack, object"
     38    exit 2
     39  case -all
     40    set mkgpc1       = 0
     41    set initdb       = 0
     42    set initscratch  = 0
     43    set initbatch    = 0
     44    set camqueue     = 0
     45    set cambatch     = 0
     46    set stackqueue   = 0
     47    set stackbatch   = 0
     48    set objectqueue  = 0
     49    set objectbatch  = 0
     50    breaksw;
    2951  case -mkgpc1
    3052    set mkgpc1 = 0
    … …  
    5274    breaksw;
    5375  default:
    54     set args=($args $1);
     76    set args=($args $targ[1]);
    5577    breaksw;
    5678  endsw
    57   shift
     79  shift targ
    5880end
     81
     82set targ = ($args)
     83set args = ""
     84
     85while ($#targ)
     86 switch ("$targ[1]")
     87  case +all
     88    set mkgpc1       = 1
     89    set initdb       = 1
     90    set initscratch  = 1
     91    set initbatch    = 1
     92    set camqueue     = 1
     93    set cambatch     = 1
     94    set stackqueue   = 1
     95    set stackbatch   = 1
     96    set objectqueue  = 1
     97    set objectbatch  = 1
     98    breaksw;
     99  case +mkgpc1
     100    set mkgpc1 = 1
     101    breaksw;
     102  case +initdb
     103    set initdb = 1
     104    breaksw;
     105  case +initscratch
     106    set initscratch = 1
     107    breaksw;
     108  case +initbatch
     109    set initbatch = 1
     110    breaksw;
     111  case "+camera"
     112    set camqueue = 1
     113    set cambatch = 1
     114    breaksw;
     115  case +stack
     116    set stackqueue = 1
     117    set stackbatch = 1
     118    breaksw;
     119  case +object
     120    set objectqueue = 1
     121    set objectbatch = 1
     122    breaksw;
     123  default:
     124    set args=($args $targ[1]);
     125    breaksw;
     126  endsw
     127  shift targ
     128end
     129
     130if ($args[1] != "") then
     131    echo "USAGE: fulltest.sh [-XX] [+XX]"
     132    echo " -XX : turn off features"
     133    echo " +XX : turn on features"
     134    echo "  features: all, mkgpc1, initdb, initscratch, initbatch, camera, stack, object"
     135    exit 2
     136endif
    59137
    60138set stackqueuename = 0

  • branches/eam_branches/ipp-20130419/ippToPsps/test/mkgpc1data.dvo

    r35417 r35554  
    3737macro mkfull
    3838  init.db
    39   mkcatdir.cam PS1_V3 PS1_V4
    40   mkcatdir.stk PS1_V3 PS1_V4
     39  # mkcatdir.cam PS1_V3 PS1_V4
     40  # mkcatdir.stk PS1_V3 PS1_V4
     41  mkcatdir.cam PS1_V4 PS1_V4
     42  mkcatdir.stk PS1_V4 PS1_V4
    4143  insert.stack.set
    4244end

  • branches/eam_branches/ipp-20130419/psconfig

    • Property svn:mergeinfo changed (with no actual effect on merging)

  • branches/eam_branches/ipp-20130419/psphot

  • branches/eam_branches/ipp-20130419/psphot/src

  • branches/eam_branches/ipp-20130419/psphot/src/psphotStackImageLoop.c

    • Property svn:mergeinfo changed (with no actual effect on merging)

  • branches/eam_branches/ipp-20130419/psvideophot

    • Property svn:mergeinfo changed (with no actual effect on merging)
Note: See TracChangeset for help on using the changeset viewer.