Index: trunk/pstamp/doc/psnotes.txt =================================================================== --- trunk/pstamp/doc/psnotes.txt (revision 20256) +++ trunk/pstamp/doc/psnotes.txt (revision 20257) @@ -6,7 +6,7 @@ This note describes the installation and operation of the IPP Postage Stamp Server. We assume that IPP source for examination and that the user has the standard -ipp configuration environment set up. - -This component of the IPP system is not intended for general distribution to ipp users +IPP configuration environment set up. + +This component of the IPP system is not intended for general distribution to IPP users and does not have streamlined installation procedures at this time. @@ -91,5 +91,5 @@ -The ipp includes a number of tools for accessing a data store. An introduction to these tools may +The IPP includes a number of tools for accessing a data store. An introduction to these tools may be seen by using the program perldoc. @@ -109,5 +109,5 @@ The files for the Data Store Server located in the IPP source tree in the directory -DataStoreServer (this directory may not be available in the distributed ipp +DataStoreServer (this directory may not be available in the distributed IPP tarballs). @@ -115,14 +115,28 @@ ----------------------------- -To set up the database go to the DataStoreServer/scripts and start up mysql. - -mysql> create database mydatabase; -mysql> use mydatabase; -mysql> source scripts/tabledefs.sql; - -Note: if you already have a current IPP database that you would like to use -for the Postage Stamp Request tables, when skip you can skip the create step -and use that database. - +The tables for the PSS are part of the regular IPP database tables. + +The Data Store implementation has been designed to not require the IPP for its +operation so the Data Store's tables are not created as part of the IPP database setup. + +Currently however, the PostageStamp Server implementation assumes that the data +store and postage stamp tables are in the same mysql database (this will be +fixed soon). + +To set up the database go to the directory DataStoreServer/scripts and start up mysql. + +If you do not already have a current IPP database that you would like to use +for the Postage Stamp Request tables issue the following commands + + mysql> create database mydatabase; + mysql> use mydatabase; + mysql> source scripts/tabledefs.sql; + +Now exit mysql and run the command + pxadmin -create -dbname mydatabase + +If you already have a database set up, then just issue the commands + mysql> use mydatabase; + mysql> source scripts/tabledefs.sql; Web Server Configuration @@ -170,4 +184,7 @@ ScriptAlias /ds-cgi /DATASTOREDIR/ds-cgi/ +(another sample of this configuration file may be found in + DataStoreServer/web/conf/httpd-datastore.conf + Here we have assumed that the data store cgi scripts will be installed in /DATASTOREDIR/ds-cgi and the data store data files will be stored in @@ -282,11 +299,11 @@ state enabled dbname ps_simtest - dvodb currently not used + dvodb NULL (currently not used) camera SIMTEST telescope SimScope need_magic 0 -The column dbname containes the name of the database that is used to look up images. To -create a project use pstamptool. In this case the ps_simtest is a database +The column dbname contains the name of the database that is used to look up images. +To create a project use pstamptool. In this example ps_simtest is a database refererencing images made by a simtest run. @@ -298,23 +315,47 @@ Postage Stamp requests are submitted in the form of a FITS binary table. The -format of this table is described in a document which can be found by going -to the page - - PostageStampServer - -on the IPP wiki. - - -The tables for the PSS are part of the regular IPP database -tables. The Data Store implementation has been designed to not require the IPP for it's -operation. -So the Data Store's tables are not created as part of the ipp database setup. - -However, the current PostageStamp Server implementation assumes that the data -store and postage stamp tables are in the same mysql database (this will be -fixes soon). - -As will be explained below, the request database does not necessarily have to be the -same as the image database. +format of this table is described in a document which may be found on the IPP wiki. +http://kiawe.ifa.hawaii.edu/IPPwiki/index.php/PostageStampServer + +Only the first table extension found in the request file is processed. + +Each postage stamp request file submitted to the PSS generates a "Request". +The name of the request is given by the value for the keyword REQ_NAME in +the extension's fits header. Each postage stamp request must have a unique REQ_NAME must be unique. + +To process the request the PSS parses the request file. +Each row in a request table contains a request specification. Each request specification +causes zero or more "Jobs" to be queued for processing. Each Job operates on +one input image. + +When all of the Jobs for a given Request complete, a Postage stamp results +file is created and it along with the images produced by the jobs are +registered in a fileset on the output Data Store. + +There are two job types specified by the value for the JOB_TYPE keyword in the +request specification. + + JOB_TYPE = "stamp" creates a "postage stamp" image. + JOB_TYPE = "get_image" builds a fileset containing copies of images + +There are two programs that may be used to create a postage stamp request. +pstamprequest creates a request file with a single request specification. + +For example + + pstamprequest -req_name REQ_00001 -project simtest req.fits \ + -byexp chip simtest.004.000 null \ + -pixcenter 500 500 -pixrange 100 100 + +creates a request file req.fits containing a request specification for a 100 x 100 pixel +postage stamp image centered at (500,500) in the source. +The source image is chip processed image for one of the object exposures from a simtest run. + +Another program that may be used to create a request file is located in +pstamp/test/pstamp_req_create. + +This program a text file and creates a multi-row postage stamp request. + + XXXXXXXXXXXXXXXXXXXXXX The following sections are to be completed @@ -322,13 +363,9 @@ -------------------- pantasks tasks -Database + DataStore for input and output Web interface (limited functionality prototype only) -Request files -Link to Spec - - Examples Index: trunk/pstamp/scripts/pstamp_finish.pl =================================================================== --- trunk/pstamp/scripts/pstamp_finish.pl (revision 20256) +++ trunk/pstamp/scripts/pstamp_finish.pl (revision 20257) @@ -110,4 +110,6 @@ my ($rlf, $reglist_name) = tempfile ("$out_dir/reglist.XXXX", UNLINK => !$save_temps); print $rlf "results.fits|||table|\n"; + + # XXX: this file is in the request's workdir not the out_dir my $err_file = "$out_dir/parse_error.txt"; if (-e $err_file ) { @@ -138,5 +140,5 @@ # including those that produced no jobs. # for now add an entry for rownum 1 and a phony error code. - # we've included parse_results.txt to the fileset if it exists + # we've included parse_error.txt to the fileset if it exists # my $rownum = 0; Index: trunk/pstamp/scripts/pstamp_results_file.pl =================================================================== --- trunk/pstamp/scripts/pstamp_results_file.pl (revision 20256) +++ trunk/pstamp/scripts/pstamp_results_file.pl (revision 20257) @@ -91,5 +91,4 @@ # output parameters -# { name => 'STAMP_NAME', type => '16A', writetype => TSTRING }, { name => 'OPTION_MASK',type => 'J', writetype => TULONG }, Index: trunk/pstamp/test/pstamp_req_create =================================================================== --- trunk/pstamp/test/pstamp_req_create (revision 20256) +++ trunk/pstamp/test/pstamp_req_create (revision 20257) @@ -49,5 +49,4 @@ { name => 'JOB_TYPE', type => '16A', writetype => TSTRING }, - { name => 'STAMP_NAME', type => '16A', writetype => TSTRING }, { name => 'OPTION_MASK',type => 'J', writetype => TULONG }, Index: trunk/pstamp/test/sample_pstamp.txt =================================================================== --- trunk/pstamp/test/sample_pstamp.txt (revision 20256) +++ trunk/pstamp/test/sample_pstamp.txt (revision 20257) @@ -15,8 +15,8 @@ # subsequent lines define the rows in the table -# ROWNUM PROJECT JOB_TYPE STAMP_NAME OPTION_MASK REQ_TYPE IMG_TYPE ID CLASS_ID COORD_MASK CENTER_X CENTER_Y WIDTH HEIGHT REQFILT MJD_MIN MJD_MAX -1 megacam-mops stamp null 1 byid chip 419 null 3 1500 1500 100 100 null 0 0 -2 megacam-mops stamp null 1 byid chip 419 null 3 1600 1500 100 100 null 0 0 -3 megacam-mops stamp null 1 byid chip 419 null 3 1700 1500 100 100 null 0 0 +# ROWNUM PROJECT JOB_TYPE OPTION_MASK REQ_TYPE IMG_TYPE ID CLASS_ID COORD_MASK CENTER_X CENTER_Y WIDTH HEIGHT REQFILT MJD_MIN MJD_MAX +1 megacam-mops stamp 1 byid chip 419 null 3 1500 1500 100 100 null 0 0 +2 megacam-mops stamp 1 byid chip 419 null 3 1600 1500 100 100 null 0 0 +3 megacam-mops stamp 1 byid chip 419 null 3 1700 1500 100 100 null 0 0 # the following row will create several jobs Index: trunk/pstamp/test/sample_simtest.txt =================================================================== --- trunk/pstamp/test/sample_simtest.txt (revision 20257) +++ trunk/pstamp/test/sample_simtest.txt (revision 20257) @@ -0,0 +1,31 @@ +# Sample Postage stamp request description file +# This can be parsed by the program pstamp_req_create to build a +# PS1_PSTAMP_REQUEST binary table + +# First line of data is for the extension header +# The order of keywords follows TABLE 6 of ICD +# +# Note that value for REQ_NAME may be overriden by a command line +# argument to pstamp_req_create allowing this file to be re-used. +# Blank and comment lines are ignored + +# REQ_NAME EXTVER +PSREQ00001 1 + +# subsequent lines define the rows in the table + +# ROWNUM PROJECT JOB_TYPE OPTION_MASK REQ_TYPE IMG_TYPE ID CLASS_ID COORD_MASK CENTER_X CENTER_Y WIDTH HEIGHT REQFILT MJD_MIN MJD_MAX +# select by exposure name +1 simtest stamp 1 byexp chip simtest.004.000 null 3 500 500 100 100 null 0 0 + +# select by database id this happens to select the same image a row 1 using 'chip_id' +2 simtest stamp 1 byid chip 2 null 3 600 500 100 100 null 0 0 + +# stamp from a chip processed image that resulted in diff_id = 1 sky coords for center, pixels for range +3 simtest stamp 1 bydiff chip 1 null 2 270.733 -23.7039 500 500 null 0 0 + +# ... and the corespoding warp +4 simtest stamp 1 bydiff warp 1 null 2 270.733 -23.7039 500 500 null 0 0 + +# ... and the stack +5 simtest stamp 1 bydiff stack 1 null 2 270.733 -23.7039 500 500 null 0 0