Index: branches/pap/pstamp/doc/dsinstall.txt
===================================================================
--- branches/pap/pstamp/doc/dsinstall.txt (revision 28003)
+++ branches/pap/pstamp/doc/dsinstall.txt (revision 28003)
@@ -0,0 +1,417 @@
+Installing the data store sample implementation
+XXX: This text was found lying around in my workspace. It needs to be
+checked for accuarcy
+
+Overview
+--------
+
+This note describes the installation and operation of the IPP Postage Stamp Server.
+We assume that IPP source for examination and that the user has the standard
+IPP configuration environment set up.
+
+This component of the IPP system is not intended for general distribution to IPP users
+and does not have streamlined installation procedures at this time.
+
+The PSS is a system which responds to requests to retrieve images from an IPP Image Server.
+The images may be either complete images or subsets of images known as "Postage Stamps".
+
+The internal mechanisms of the PSS are also used to service MOPS detectabiltiy
+query requests.
+
+Given this more general utility it should perhaps be called the "IPP Request Server".
+
+
+Data Store
+----------
+The PSS uses the Data Store mechanism to find incoming requests and to
+provide results to the end user.
+
+A DataStore is simply a web server with URLS that supports an interface
+that allows collections of data called File Sets to be located and accessed.
+
+A Data Store is organized into 3 levels. These are described in the following sections.
+
+Data Store Root - a list of Products
+-------------------------------------
+
+Assume that a datastore is located on the server WEBSERVERHOST.
+An HTTP get to the url http://WEBSERVERHOST/ds/index.txt would give a list of the
+products in the Data Store.
+
+ # productID |Most recent |Time registered |type |Description |
+ pstampresults|web204 |2008-10-16T00:35:16Z|psresults|Postage Stamp Results |
+ pstamprequests|TEST0012 |2008-08-29T03:26:40Z|psrequest|Postage Stamp Requests |
+ mops-pstamp-results|MOPSSTAMPREQ20080813T214918Z_00|2008-08-13T21:49:42Z|psresults|Postage Stamp Results for MOPS |
+
+Each line is divided into fields separated by a 'pipe' character | .
+
+Data Store Product - a list of filesets
+---------------------------------------
+
+Similarly the url http://WEBSERVERHOST/ds/pstampresults/index.txt gives the list of filesets
+in the product pstampresults:
+
+ # filesetID|time registered |type |
+ TEST0011 |2008-08-29T03:05:03Z|PSRESULTS|
+ TEST0012 |2008-08-29T03:31:36Z|PSRESULTS|
+ web190 |2008-10-15T02:29:11Z|PSRESULTS|
+ web191 |2008-10-15T02:32:16Z|PSRESULTS|
+ web192 |2008-10-15T02:38:50Z|PSRESULTS|
+ web193 |2008-10-15T03:08:57Z|PSRESULTS|
+ web194 |2008-10-15T23:32:11Z|PSRESULTS|
+ web195 |2008-10-15T23:35:39Z|PSRESULTS|
+ web196 |2008-10-15T23:52:40Z|PSRESULTS|
+ web197 |2008-10-15T23:59:05Z|PSRESULTS|
+ web198 |2008-10-16T00:04:17Z|PSRESULTS|
+ web200 |2008-10-16T00:17:46Z|PSRESULTS|
+ web204 |2008-10-16T00:35:16Z|PSRESULTS|
+
+Note that the fileset ID for the latest fileset (web204) in the list appears in the root listing in the last
+section.
+
+The product list protocol accepts a query string which can be used to limit the listing to all filesets
+registered after a given fileset.
+
+For example, http://WEBSERVERHOST/ds/pstampresults/index.txt?web198 yields
+
+ # filesetID|time registered |type |
+ web200 |2008-10-16T00:17:46Z|PSRESULTS|
+ web204 |2008-10-16T00:35:16Z|PSRESULTS|
+
+
+File Set - a list of files
+--------------------------
+http://ippds.somwhere.net/ds/pstampresults/web205/index.txt
+
+ # fileID |bytes |md5sum |type |
+ results.fits |28800 |4e2048479e551844e31102a301d50740|table |
+ # fileID |bytes |md5sum |type |chipname|
+ 1_1_o4748g0259o.ota01.fits|19840320|19a6233ab966cf3e13b7da8271dead67|chip |XY01 |
+ 1_2_o4748g0259o.ota02.fits|20416320|625bf8198dd7aef7c892af03584ee0ed|chip |XY02 |
+
+ .... followed by 58 other files in this GPC1 exposure
+
+
+The IPP includes a number of tools for accessing a data store. An introduction to these tools may
+be seen by using the program perldoc.
+
+ dsrootls
+ dsproductls
+ dsfilesetls
+ dsget
+
+
+Installing the IPP Data Store Server Implementation
+---------------------------------------------------
+
+The data store implementation consists of a set of cgi scripts which are
+invoked by an apache web server. The indexes for the Data Store are stored in a mysql
+database. The cgi scripts use DBI perl module to query the database. No database updates
+are performed by the Data Store cgi scripts.
+
+The files for the Data Store Server located in the IPP source tree in the directory
+DataStoreServer (this directory may not be available in the distributed IPP
+tarballs).
+
+Mysql data base configuration
+-----------------------------
+
+The tables for the PSS are part of the regular IPP database tables.
+
+The Data Store implementation has been designed to not require the IPP for its
+operation so the Data Store's tables are not created as part of the IPP database setup.
+
+Currently however, the PostageStamp Server implementation assumes that the data
+store and postage stamp tables are in the same mysql database (this will be
+fixed soon).
+
+To set up the database go to the directory DataStoreServer/scripts and start up mysql.
+
+If you do not already have a current IPP database that you would like to use
+for the Postage Stamp Request tables issue the following commands
+
+ mysql> create database mydatabase;
+ mysql> use mydatabase;
+ mysql> source scripts/tabledefs.sql;
+
+Now exit mysql and run the command
+ pxadmin -create -dbname mydatabase
+
+If you already have a database set up, then just issue the commands
+ mysql> use mydatabase;
+ mysql> source scripts/tabledefs.sql;
+
+Web Server Configuration
+------------------------
+The apache configuration needs to have the following entries added to
+the appropriate configuration file. (Different apache distributions have
+different locations for these files so it's up to the installer to
+figure out where these configuration lines should go).
+
+
+
+ DirectoryIndex /ds-cgi/dsindex.cgi
+
+ #Options None
+
+ # turn on server side includes
+ Options +Includes
+
+ # files will be processed for server side includes if they have the
+ # executable bit set
+ XBitHack on
+
+ AddType text/plain .txt
+ AddOutputFilter INCLUDES .txt
+
+ AllowOverride None
+
+ Order allow,deny
+ Allow from all
+
+
+ #
+ # Location of the cgi scripts
+ #
+
+
+ Options ExecCGI
+
+
+ Order allow,deny
+ Allow from all
+
+
+ Alias /ds /DATASTOREDIR/dsroot/
+ ScriptAlias /ds-cgi /DATASTOREDIR/ds-cgi/
+
+(another sample of this configuration file may be found in
+ DataStoreServer/web/conf/httpd-datastore.conf
+
+Here we have assumed that the data store cgi scripts will be installed in
+/DATASTOREDIR/ds-cgi and the data store data files will be stored in
+/DATASTOREDIR/dsroot.
+
+To install the cgi scripts
+ cd your_ipp_src_directory
+ cd DataStoreServer
+ scripts/installscripts --dsroot /DATASTOREDIR/dsroot --cgidir /DATASTOREDIR/ds-cgi
+
+The file /DATASTOREDIR/ds-cgi/dsshellconfig is a shell script that sets
+several environment variables that are required by the scripts.
+
+This script should be editied to customize the values for the installation.
+(This file will not be overwritten by subsequent invocations of
+installscripts).
+
+There are some scripts that the Postage Stamp Server uses that need to be
+installed into the IPP build directory. These are not included in the current IPP
+build. To build them invoke the following commands from the DataStoreServer
+directory.
+ psautogen
+ make install
+
+
+Testing the DataStore
+---------------------
+At this point an empty data store should be up and running. This can be tested
+by going to the following url in a web browser
+ http://WEBSERVERHOST/ds/index.txt
+
+If things are working properlly you should see the following output
+
+# productID |Most recent |Time registered |type |Description |
+
+If an errror occurs check the web server error log and begin debugging.
+
+The following command may be used to create a test product in the data store
+
+dsprodtool --dbname mydatabase --dsroot /DATASTOREDIR/dsroot/ \
+ --add testproduct --type psresults --description 'test data store product'
+
+Now if you point to http://WEBSERVERHOST/ds/index.txt with a web browser you
+should see
+
+# productID |Most recent |Time registered |type |Description |
+testproduct |none |2008-10-16T20:48:23Z|psresults|test data store product |
+
+(of course your timestamp will be different).
+
+As a final test we will add a fileset to the data store.
+
+echo this is a file > testfile.txt
+
+echo 'testfile.txt|||text|' | dsreg --dbname mydatabase --dsroot /DATASTOREDIR/dsroot/ \
+ --product testproduct --add testfileset --type PSRESULTS --list - --copy --datapath .
+
+-list - tells dsreg to get the list of files for the fileset from stdin.
+(Usually a file is used). The input to the command gives the name of the file to be added to the
+fileset, blank entries for size and md5sum, and the file type 'text'.
+
+Now if you go to http://WEBSERVERHOST/ds/testfileset/index.txt you should see
+the fileset.
+
+If you leave off the index.txt in the urls you will get a more web browser
+friendly html pages with links.
+For example
+ http://WEBSERVERHOST/ds
+
+To delete the test product
+ dsprodtool --dbname mydatabase --dsroot /DATASTOREDIR/dsroot/ --del psresults --rm
+
+Set up the product for the postage stamp results with
+
+ dsprodtool --dbname mydatabase --dsroot /DATASTOREDIR/dsroot/ \
+ --add pstampresults --type psresults --description 'Postage Stamp results'
+
+and a product for requests
+
+ dsprodtool --dbname mydatabase --dsroot /DATASTOREDIR/dsroot/ \
+ --add pstamprequests --type psrequest --description 'Postage Stamp requests'
+
+Postage Stamp Server Set up
+---------------------------
+
+Now we are ready to set up the Postage Stamp Server.
+
+The programs that make up the PSS are controlled by a set of pantasks tasks.
+There are some configuration variables that must be added to the site.config file.
+
+ # root directory of the data store
+ # this must match the directory used above in the Data Store setup
+ DATA_STORE_ROOT STR /DATASTOREDIR/dsroot
+
+ # default data store product used for results
+ # (used by the web interface)
+ PSTAMP_DATA_STORE_PRODUCT STR pstampresults
+
+ # This directory is used for downloaded request files and other purposes
+ # it can go anywhere. The permissions on the directory must be such that
+ # that the web server program has permission to write there.
+ PSTAMP_WORKDIR STR /SOMEWHERE/pstamp-work
+
+PSS projects
+------------
+Each Postage stamp request selects images from a particular 'project'.
+Each project has a row in the table pstampProject has the following columns
+
+ column example
+ -------------------
+ name simtest
+ state enabled
+ dbname ps_simtest
+ dvodb NULL (currently not used)
+ camera SIMTEST
+ telescope SimScope
+ need_magic 0
+
+The column dbname contains the name of the database that is used to look up images.
+To create a project use pstamptool. In this example ps_simtest is a database
+refererencing images made by a simtest run.
+
+pstamptool -dbname mydatabase -addproject -name foo -imagedb ps_simtest -inst SIMTEST -telescope SimScope
+
+
+Postage Stamp Request Tables
+----------------------------
+
+Postage Stamp requests are submitted in the form of a FITS binary table. The
+format of this table is described in a document which may be found on the IPP wiki.
+http://kiawe.ifa.hawaii.edu/IPPwiki/index.php/PostageStampServer
+
+Only the first table extension found in the request file is processed.
+
+Each postage stamp request file submitted to the PSS generates a "Request".
+The name of the request is given by the value for the keyword REQ_NAME in
+the extension's fits header. Each postage stamp request must have a unique REQ_NAME must be unique.
+
+To process the request the PSS parses the request file.
+Each row in a request table contains a request specification. Each request specification
+causes zero or more "Jobs" to be queued for processing. Each Job operates on
+one input image.
+
+When all of the Jobs for a given Request complete, a Postage stamp results
+file is created and it along with the images produced by the jobs are
+registered in a fileset on the output Data Store.
+
+There are two job types specified by the value for the JOB_TYPE keyword in the
+request specification.
+
+ JOB_TYPE = "stamp" creates a "postage stamp" image.
+ JOB_TYPE = "get_image" builds a fileset containing copies of images
+
+There are two programs that may be used to create a postage stamp request.
+pstamprequest creates a request file with a single request specification.
+
+For example
+
+ pstamprequest -req_name REQ_00001 -project simtest /mydirectory/req.fits \
+ -byexp chip simtest.004.000 null \
+ -pixcenter 500 500 -pixrange 100 100
+
+creates a request file req.fits containing a request specification for a 100 x 100 pixel
+postage stamp image centered at (500,500) in the source.
+The source image is chip processed image for one of the object exposures from a simtest run.
+
+Another program that may be used to create a request file is located in
+pstamp/test/pstamp_req_create. This program takes a text file and creates a multi-row postage
+stamp request. A couple of example input files may be found in pstamp/test/*.txt
+
+Queueing a PostageStamp Request
+-------------------------------
+The postage stamp server looks for work to do by examining a collection of
+input Data Stores. For example, the MOPS team will have a datastore onto which
+they will submit their postage stamp requests.
+
+There will also be a web based interface. A prototype for this will be
+discussed in a later section of this document.
+
+For testing, we may queue a request by hand using the program pstamptool
+
+ pstamptool -dbname mydatabase -addreq -uri /mydirectory/req.fits
+
+This adds an entry to the table pstampRequests with state = 'new'
+
+Request Processing
+------------------
+
+Postage stamp requests are processed by a set of pantasks tasks.
+
+ Task Status
+ AV Name Njobs Ngood Nfail Ntime Command
+ ++ pstamp.request.find 921 921 0 0 pstamp_queue_requests.pl
+ ++ pstamp.request.run 13 13 0 0 pstamp_parser_run.pl
+ ++ request.finish.load 1839 1839 0 0 pstamptool
+ ++ request.finish.run 14 14 0 0 request_finish.pl
+ ++ pstamp.job.load 1840 1840 0 0 pstamptool
+ ++ pstamp.job.run 30 30 0 0 pstamp_job_run.pl
+ ++ pstamp.request.load 1566 1566 0 0 pstamptool
+
+pstamp.request.find examines incoming data stores for new requests and adds
+them if found
+
+pstamp.request.load dispatches pending requests for parsing
+
+pstamp.request.run runs the request parser which queues jobs to service the
+request specifications.
+
+pstamp.job.load dispatches pending jobs for processing
+
+pstamp.job.run runs a single job
+
+pstamp.finish.load looks for requests which have no pending jobs
+
+pstamp.finish.run builds the results fileset and registers it on the ouptut
+Data Store.
+
+Output Fileset
+-------------
+A successful request will generate a fileset with the name the REQ_NAME of the
+request in the output Data Store. The output product is determined by the
+source of the request as discussed in the next section. (If the request did
+not come from a data store the product is given by the value of PSTAMP_DATA_STORE_PRODUCT
+in the IPP site.config file.
+
+
+
+
Index: branches/pap/pstamp/scripts/Makefile.am
===================================================================
--- branches/pap/pstamp/scripts/Makefile.am (revision 27708)
+++ branches/pap/pstamp/scripts/Makefile.am (revision 28003)
@@ -3,4 +3,5 @@
install_files = \
+ pstamp_cleanup.pl \
pstamp_finish.pl \
pstamp_insert_request.pl \
@@ -20,4 +21,5 @@
pstamp_checkdependent.pl \
request_finish.pl \
+ detectability_respond.pl \
detect_query_read \
detect_response_create \
Index: branches/pap/pstamp/scripts/detect_query_read
===================================================================
--- branches/pap/pstamp/scripts/detect_query_read (revision 27708)
+++ branches/pap/pstamp/scripts/detect_query_read (revision 28003)
@@ -78,10 +78,16 @@
comment => 'site identifier (MPC observatory code)',
value => undef
- }
+ },
+ 'STAGE' => {
+ name => 'STAGE',
+ writetype => TSTRING,
+ comment => 'processing stage to examine',
+ value => undef
+ }
};
# key_array insures that the order that the keywords is printed out is
# the same as the ICD
-my @key_array = qw( QUERY_ID FPA_ID MJD_OBS FILTER OBSCODE );
+my @key_array = qw( QUERY_ID FPA_ID MJD_OBS FILTER OBSCODE STAGE);
# Specification of columns
@@ -144,4 +150,7 @@
my $name = $header->{$key}->{name};
my $value = $inHeader->{$name};
+ if (($key eq 'STAGE')&& !(defined($value))) {
+ $value = 'DIFF';
+ }
# get rid of quotes and whitespace
$value =~ s/\'//g;
Index: branches/pap/pstamp/scripts/detectability_respond.pl
===================================================================
--- branches/pap/pstamp/scripts/detectability_respond.pl (revision 27708)
+++ branches/pap/pstamp/scripts/detectability_respond.pl (revision 28003)
@@ -41,10 +41,12 @@
my $EXTVER = 1.0;
my $EXTNAME = 'MOPS_DETECTABILITY_RESPONSE';
-my ($req_id,$req_name,$product,$need_magic,$missing_tools);
-my ($request_file,$output,$dbname,$verbose,$save_temps);
+my ($req_id,$req_name,$product,$need_magic,$missing_tools,$project);
+my ($request_file,$output,$workdir,$dbname,$dbserver,$verbose,$save_temps);
GetOptions(
- 'input=s' => \$request_file,
+ 'input=s' => \$request_file,
'output=s' => \$output,
+ 'workdir=s' => \$workdir,
'dbname=s' => \$dbname,
+ 'dbserver=s' => \$dbserver,
'verbose' => \$verbose,
'save-temps' => \$save_temps,
@@ -55,5 +57,5 @@
-exitval => 3,
) unless
- defined $request_file and defined $output and defined $dbname;
+ defined $request_file and defined $output and defined $workdir and defined $dbname;
my $detect_query_read = can_run('detect_query_read') or (warn "Can't find detect_query_read" and $missing_tools = 1);
@@ -71,15 +73,26 @@
my $ipprc = PS::IPP::Config->new();
-#my $tmp_dir = "/data/${host}.0/tmp/";
+
+if (!$dbserver) {
+ $dbserver = metadataLookupStr($ipprc->{_siteConfig}, 'PS_DBSERVER');
+}
+# This is hardcoded in for the moment.
+$project = resolve_project($ipprc,"gpc1",$dbname,$dbserver);
+my $imagedb = $project->{dbname};
+if (!$imagedb) {
+ carp("failed to find imagedb for project: $project");
+}
#
# Parse input request file using detect_query_read (as it's already written).
#
-
my $dqr_command = "$detect_query_read --input $request_file";
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
run(command => $dqr_command, verbose => $verbose);
unless ($success) {
- my_die("Unable to perform $dqr_command error code: $error_code",-1,-1,-1,-1,-1,-1);
+ # This is a problem, because I'm not sure how we handle a failure to read something.
+ # We need to return a $PSTAMP_INVALID_REQUEST, I think, but if we can't read it,
+ # we can't send that response back.
+ die("Unable to perform $dqr_command error code: $error_code");
}
my %query = ();
@@ -119,6 +132,6 @@
my %image_list_hash;
for (my $i = 1; $i < $Nrows; $i++) {
-# print "$i $Nrows $query{CONTENT}{RA1_DEG}[$i] $query{CONTENT}{DEC1_DEG}[$i]\n";
-
+ # This could use the fact that locate_images now accepts position arrays, but
+ # I'll save that for after I get the majority of things working.
my $image_set_tmp = find_image_set($query{HEADER}{FPA_ID}[0],$query{HEADER}{STAGE}[0],
$query{HEADER}{MJD_OBS}[0],$query{HEADER}{FILTER}[0],
@@ -126,10 +139,21 @@
$query{CONTENT}{ROWNUM}[$i],$verbose);
unless (%$image_set_tmp) {
- my_die("No images were returned.",$query{HEADER}{QUERY_ID}[0],$query{HEADER}{FPA_ID}[0],
- $query{HEADER}{MJD_OBS}[0],$query{HEADER}{FILTER}[0],$query{HEADER}{OBSCODE}[0],
- $query{HEADER}{STAGE}[0]);
- }
- print "=== $image_set_tmp->{IMAGE}\n $image_set_tmp->{PSF}\n $image_set_tmp->{MASK}\n $image_set_tmp->{WEIGHT}\n $image_set_tmp->{SKY_COORDINATES}\n $image_set_tmp->{ROWNUM}\n";
- # This appends, assuming that if we get an image, we also get the identical psf/mask/weight/etc.
+ # No images were returned, so create a dummy entry that
+ $image_list_hash{'no_image'}{IMAGE} = 'no_image';
+ $image_list_hash{'no_image'}{PSF} = 'no_psf';
+ $image_list_hash{'no_image'}{MASK} = 'no_mask';
+ $image_list_hash{'no_image'}{WEIGHT} = 'no_weight';
+ $image_list_hash{'no_image'}{CATALOG} = 'no_catalog';
+ $image_list_hash{'no_image'}{CLASS_ID} = 'no_class';
+ $image_list_hash{'no_image'}{ERROR} = $PSTAMP_NO_IMAGE_MATCH;
+ push @{ $image_list_hash{'no_image'}{SKY_COORDINATES} }, "$query{CONTENT}{RA1_DEG}[$i] $query{CONTENT}{DEC1_DEG}[$i]";
+ push @{ $image_list_hash{'no_image'}{ROWNUM} }, $query{CONTENT}{ROWNUM}[$i];
+ next;
+ }
+# print "=== $image_set_tmp->{IMAGE}\n $image_set_tmp->{PSF}\n";
+# print " $image_set_tmp->{MASK}\n $image_set_tmp->{WEIGHT}\n";
+# print " $image_set_tmp->{SKY_COORDINATES}\n $image_set_tmp->{ROWNUM}\n";
+
+ # This indexes the results for identical images into the same hash.
$image_list_hash{$image_set_tmp->{IMAGE}}{IMAGE} = $image_set_tmp->{IMAGE};
$image_list_hash{$image_set_tmp->{IMAGE}}{PSF} = $image_set_tmp->{PSF};
@@ -138,14 +162,19 @@
$image_list_hash{$image_set_tmp->{IMAGE}}{CATALOG} = $image_set_tmp->{CATALOG};
$image_list_hash{$image_set_tmp->{IMAGE}}{CLASS_ID} = $image_set_tmp->{CLASS_ID};
+ $image_list_hash{$image_set_tmp->{IMAGE}}{ERROR} = $image_set_tmp->{ERROR};
push @{ $image_list_hash{$image_set_tmp->{IMAGE}}{SKY_COORDINATES} }, $image_set_tmp->{SKY_COORDINATES};
push @{ $image_list_hash{$image_set_tmp->{IMAGE}}{ROWNUM} }, $image_set_tmp->{ROWNUM};
}
+
my $i = 0;
-
foreach my $k (keys %image_list_hash) {
+ # If we errored out on finding an image, we need to not try to run psphot here.
+ if ($image_list_hash{$k}{ERROR} != 0) {
+ next;
+ }
# Write coordinates of the requested targets to a file.
- my ($coordfile,$coordname) = tempfile("/tmp/detect.coords.$i.XXXX",
+ my ($coordfile,$coordname) = tempfile("${workdir}/detect.coords.$i.XXXX",
UNLINK => !$save_temps);
- my ($targetfile,$targetname) = tempfile("/tmp/detect.targets.$i.XXXX",
+ my ($targetfile,$targetname) = tempfile("${workdir}/detect.targets.$i.XXXX",
UNLINK => !$save_temps);
@@ -153,5 +182,5 @@
print $coordfile "$image_list_hash{$k}{SKY_COORDINATES}[$j]\n";
}
-
+# print "$k\n";
# Convert the sky coordinates to image coordinates with ppCoord.
my $command = "ppCoord -astrom $image_list_hash{$k}{CATALOG} -radec $coordname";
@@ -171,7 +200,7 @@
}
-
+# print "psphot $image_list_hash{$k}{PSF}\n";
# Run psphotForced on the target list.
- my $tmpdir = tempdir("detect.$i.XXXX", DIR => "/tmp/", CLEANUP => !$save_temps);
+ my $tmpdir = tempdir("detect.$i.XXXX", DIR => "${workdir}/", CLEANUP => !$save_temps);
$image_list_hash{$k}{OUTROOT} = "$tmpdir/detectability.$query{HEADER}{STAGE}[0].$query{HEADER}{FPA_ID}[0]";
@@ -185,8 +214,5 @@
run(command => $psphot_cmd, verbose => $verbose);
unless ($success) {
- my_die("Unable to perform $psphot_cmd. Error_code: $error_code",
- $query{HEADER}{QUERY_ID}[0],$query{HEADER}{FPA_ID}[0],
- $query{HEADER}{MJD_OBS}[0],$query{HEADER}{FILTER}[0],$query{HEADER}{OBSCODE}[0],
- $query{HEADER}{STAGE}[0],$error_code);
+ $image_list_hash{$k}{ERROR} = $PSTAMP_SYSTEM_ERROR;
}
}
@@ -196,16 +222,28 @@
#
my @rownums = ();
+my @out_errors = ();
my @psphot_Npix = ();
my @psphot_Qfact= ();
my @psphot_flux = ();
+
foreach my $k (keys %image_list_hash) {
- my $cmf = "$image_list_hash{$k}{OUTROOT}.$image_list_hash{$k}{CLASS_ID}.cmf";
-
- my ($tmp_Npix,$tmp_Qfact,$tmp_flux) = read_cmf_file($cmf,$image_list_hash{$k}{EXTENSION_BASE});
-
- push @rownums, @{ $image_list_hash{$k}{ROWNUM} };
- push @psphot_Npix, @{ $tmp_Npix };
- push @psphot_Qfact, @{ $tmp_Qfact };
- push @psphot_flux, @{ $tmp_flux };
+ if ($image_list_hash{$k}{ERROR} == 0) {
+ my $cmf = "$image_list_hash{$k}{OUTROOT}.$image_list_hash{$k}{CLASS_ID}.cmf";
+
+ my ($tmp_Npix,$tmp_Qfact,$tmp_flux) = read_cmf_file($cmf,$image_list_hash{$k}{EXTENSION_BASE});
+
+ push @rownums, @{ $image_list_hash{$k}{ROWNUM} };
+ push @out_errors, (map { $image_list_hash{$k}{ERROR} } @{ $image_list_hash{$k}{ROWNUM} });
+ push @psphot_Npix, @{ $tmp_Npix };
+ push @psphot_Qfact, @{ $tmp_Qfact };
+ push @psphot_flux, @{ $tmp_flux };
+ }
+ else {
+ push @rownums, @{ $image_list_hash{$k}{ROWNUM} };
+ push @out_errors, (map { $image_list_hash{$k}{ERROR} } @{ $image_list_hash{$k}{ROWNUM} });
+ push @psphot_Npix, (map { 0 } @{ $image_list_hash{$k}{ROWNUM} });
+ push @psphot_Qfact, (map { 0.0 } @{ $image_list_hash{$k}{ROWNUM} });
+ push @psphot_flux, (map { 0.0 } @{ $image_list_hash{$k}{ROWNUM} });
+ }
}
@@ -214,20 +252,15 @@
$query{HEADER}{MJD_OBS}[0],$query{HEADER}{filter}[0],
$query{HEADER}{obscode}[0],
- \@rownums, \@psphot_Npix, \@psphot_Qfact, \@psphot_flux);
-
+ \@rownums, \@out_errors, \@psphot_Npix, \@psphot_Qfact, \@psphot_flux);
+# print "Wrote response file $output\n";
#
# Add to datastore
#
-# my $finish_command = "dquery_finish.pl --req_id $req_id --req_name $req_name --product $product";
-# ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
-# run(command => $finish_command, verbose => $verbose);
-# unless ($success) {
-# warn ("Unable to perform $finish_command error code: $error_code");
-# }
-
+# Files are added to the datastore by dquery_finish.pl
#
# Cleanup
#
# Since everything is written to temporary files, there should be nothing to cleanup.
+
#
# Utilities
@@ -242,5 +275,6 @@
my $index = shift;
my $verbose = shift;
-
+
+ # This is the set of things that we need in order to run psphotForced
my $option_mask |= 1;
$option_mask |= $PSTAMP_SELECT_IMAGE;
@@ -252,8 +286,21 @@
my $mjd_max = $mjd + 1;
+ # Construct a row list.
+ my @rowList;
+ $rowList[0]->{CENTER_X} = $ra;
+ $rowList[0]->{CENTER_Y} = $dec;
+ $rowList[0]->{ID} = 1;
+ $rowList[0]->{STAGE} = $stage;
+ $rowList[0]->{COORD_MASK} = 0;
+
+ # print "$stage\n";
# Call the PStamp code to find the images that contain the target on the given MJD in the specified filter.
- my @images = locate_images($ipprc,$dbname,"bycoord",$stage,
- undef,undef,undef,$option_mask,$need_magic,
- $ra,$dec,$mjd_min,$mjd_max,$filter . ".00000",undef,$verbose);
+ my @images = locate_images($ipprc,$imagedb,
+ \@rowList,
+ "bycoord",$stage,
+ undef,undef,undef,
+ $option_mask,$need_magic,
+ # $ra,$dec,
+ $mjd_min,$mjd_max,$filter . ".00000",undef,$verbose);
my %image_info = ();
@@ -270,7 +317,5 @@
}
elsif ($stage eq 'stack') {
- # Stacks hide the exposure name very well, so I'm
- # choosing the stack_id as the FPA_ID to match. This
- # probably needs to be approved somehow.
+ # Stacks hide the exposure name very well, so we can only match against stage_id
if (${ $j }{stage_id} ne $FPA_ID) {
next;
@@ -285,8 +330,43 @@
}
# Debug prints of all the components of this image
- foreach my $k (keys %{ $j }) {
- print "$i $j $k ${ $j }{$k}\n";
- }
-
+# foreach my $k (keys %{ $j }) {
+# if ($k eq 'row_index') {
+# print "$i $j $k @{${ $j }{$k} }\n";
+# }
+# print "$i $j $k ${ $j }{$k}\n";
+# }
+
+ # Check for existance of the images. Drawn mostly from pstampparse.pl
+ my $run_state = ${ $j }{state};
+ my $data_state = ${ $j }{data_state};
+ $data_state = $run_state if $stage eq 'stack';
+ my $fault = 0;
+ if (($run_state eq 'goto_purged') or ($data_state eq 'purged') or
+ ($run_state eq 'drop') or
+ ($run_state eq 'error_cleaned') or
+ ($run_state eq 'goto_scrubbed') or ($data_state eq 'scrubbed')) {
+ # image is gone and it's not coming back
+ $fault = $PSTAMP_GONE;
+ }
+ elsif (($data_state ne 'full') or ($need_magic and (${ $j }{magicked} < 0))) {
+ if (($stage eq 'stack')||($stage eq 'diff')) {
+ # updating stacks and diffs isn't implemented
+ $fault = $PSTAMP_NOT_IMPLEMENTED;
+ }
+ if ($stage eq 'chip') {
+ my $burntool_state = ${ $j }{burntool_state};
+ if ($burntool_state and (abs($burntool_state) < 14)) {
+ $fault = $PSTAMP_NOT_AVAILABLE;
+ }
+ }
+
+ if ($fault == 0) {
+ # This bombs us out to dqueryparse, which will then flag a job for this run to be updated.
+ my_die_for_update($data_state,$query{HEADER}{STAGE}[0],
+ ${ $j }{stage_id},${ $j }{class_id} || ${ $j }{skycell_id},
+ $need_magic,$imagedb,$PSTAMP_NOT_AVAILABLE);
+ }
+ }
+
# This image matches, so we want to save the information into our output structure
$image_info{ROWNUM} = $index;
@@ -295,4 +375,5 @@
$image_info{MASK} = ${ $j }{mask};
$image_info{WEIGHT} = ${ $j }{weight};
+ $image_info{ERROR} = $fault;
$image_info{SKY_COORDINATES} = "$ra $dec";
# To do sky->image coordinate transformations, we need to use the cmf/smf file. If
@@ -405,4 +486,5 @@
my $obscode = shift;
my $rownum_ref = shift;
+ my $out_err_ref = shift;
my $psphot_Npix_ref = shift;
my $psphot_Qfact_ref = shift;
@@ -413,5 +495,7 @@
my $columns = [
# matching rownum from detectability original request
- { name => 'ROWNUM', type => '20A', writetype => TSTRING },
+ { name => 'ROWNUM', type => 'V', writetype => TULONG },
+ # any errors that occurred during processing
+ { name => 'ERROR_CODE', type => 'V', writetype => TULONG },
# number of pixels used in hypothetical PSF for the query detection
{ name => 'DETECT_N', type => 'V', writetype => TULONG },
@@ -455,4 +539,5 @@
for (my $i = 0; $i < $numRows; $i++) {
push @{$colData{'ROWNUM'}}, ${ $rownum_ref }[$i];
+ push @{$colData{'ERROR_CODE'}}, ${ $out_err_ref }[$i];
push @{$colData{'DETECT_N'}}, ${ $psphot_Npix_ref }[$i];
push @{$colData{'DETECT_F'}}, ${ $psphot_Qfact_ref }[$i];
@@ -529,2 +614,16 @@
}
+
+sub my_die_for_update {
+ my $state = shift;
+ my $stage = shift;
+ my $stage_id = shift;
+ my $component = shift;
+ my $need_magic = shift;
+ my $imagedb = shift;
+ my $exit_code = shift;
+
+ print "$state $stage $stage_id $component $need_magic $imagedb\n";
+ print STDERR "$state $stage $stage_id $component $need_magic $imagedb\n";
+ exit($exit_code);
+}
Index: branches/pap/pstamp/scripts/dquery_finish.pl
===================================================================
--- branches/pap/pstamp/scripts/dquery_finish.pl (revision 27708)
+++ branches/pap/pstamp/scripts/dquery_finish.pl (revision 28003)
@@ -20,5 +20,5 @@
use PS::IPP::Config qw( :standard );
-my ( $req_id, $req_name, $req_file, $product, $out_dir, $dbname, $dbserver, $verbose, $save_temps );
+my ( $req_id, $req_name, $req_file, $product, $outdir, $dbname, $dbserver, $verbose, $save_temps );
GetOptions(
@@ -27,5 +27,5 @@
'req_file=s' => \$req_file,
'product=s' => \$product,
- 'out_dir=s' => \$out_dir,
+ 'outdir=s' => \$outdir,
'dbname=s' => \$dbname,
'dbserver=s' => \$dbserver,
@@ -41,5 +41,5 @@
$err .= "--req_name is required\n" if !$req_name;
$err .= "--product is required\n" if !$product;
-$err .= "--out_dir is required\n" if !$out_dir;
+$err .= "--outdir is required\n" if !$outdir;
die "$err" if $err;
@@ -76,16 +76,16 @@
die "product directory does not exist $prod_dir";
}
-my $out_dir = "$prod_dir/$req_name";
+my $outdir = "$prod_dir/$req_name";
}
-if (! -e $out_dir ) {
+if (! -e $outdir ) {
# something must have gone wrong at the parse stage
- print STDERR "output fileset directory $out_dir does not exist\n" if $verbose;
- if (! mkdir $out_dir ) {
+ print STDERR "output fileset directory $outdir does not exist\n" if $verbose;
+ if (! mkdir $outdir ) {
stop_request($req_id, $PS_EXIT_SYS_ERROR, $verbose);
- die "cannot create output directory $out_dir";
+ die "cannot create output directory $outdir";
}
-} elsif (! -d $out_dir) {
+} elsif (! -d $outdir) {
stop_request($req_id, $PS_EXIT_SYS_ERROR, $verbose);
- die "output fileset directory $out_dir exists but is not a directory";
+ die "output fileset directory $outdir exists but is not a directory";
}
@@ -118,13 +118,20 @@
# XXX: have the jobs produce the reglist as with postage stamp requests
-my ($REGLIST, $reg_list) = tempfile("$out_dir/reqlist.XXXX", UNLINK => !$save_temps);
+my ($REGLIST, $reg_list) = tempfile("$outdir/reqlist.XXXX", UNLINK => !$save_temps);
+
foreach my $job (@jobs) {
my $job_id = $job->{job_id};
my $response_file = "response${job_id}.fits";
- my $response_path = "$out_dir/$response_file";
+ my $response_path = "$outdir/$response_file";
if (-e $response_path) {
# the job generated a response file put it into the Data Store
print $REGLIST "$response_file|||table|\n";
+
+ # do the same if we have an error file. Should the parse data be uploaded as well?
+ my $err_file = "parse_error.txt";
+ if (-e "$outdir/$err_file") {
+ print $REGLIST "$err_file|||text|\n";
+ }
} else {
print STDERR "detect_query response file for job $job_id not found\n" if $verbose;
@@ -136,6 +143,7 @@
if (-s $reg_list) {
my $command = "$dsreg --add $req_name --product $product --list $reg_list";
- $command .= " --copy --datapath $out_dir";
+ $command .= " --copy --datapath $outdir";
$command .= " --type MOPS_DETECTABILITY_RESPONSE";
+ $command .= " --ps0 $req_id";
$command .= " --dbname $dbname" if $dbname;
@@ -158,6 +166,6 @@
my $verbose = shift;
- my $command = "$pstamptool -updatereq -req_id $req_id -state stop";
- $command .= " -fault $fault" if $fault;
+ my $command = "$pstamptool -updatereq -req_id $req_id -set_state stop";
+ $command .= " -set_fault $fault" if $fault;
$command .= " -dbname $dbname" if $dbname;
$command .= " -dbserver $dbserver" if $dbserver;
Index: branches/pap/pstamp/scripts/dqueryparse.pl
===================================================================
--- branches/pap/pstamp/scripts/dqueryparse.pl (revision 27708)
+++ branches/pap/pstamp/scripts/dqueryparse.pl (revision 28003)
@@ -2,16 +2,15 @@
#
# parse a MOPS_DETCTABILITY_QUERY table and create a results file
-#
-# Note: this file is currently only a placeholder which creates a fake response file
-# and adds a completed job to the database
#
use strict;
use warnings;
-
+use Carp;
use Getopt::Long qw( GetOptions );
use Pod::Usage qw( pod2usage );
use IPC::Cmd 0.36 qw( can_run run );
+use PS::IPP::PStamp::RequestFile qw( :standard );
+use PS::IPP::PStamp::Job qw( :standard );
use PS::IPP::Config qw($PS_EXIT_SUCCESS
$PS_EXIT_UNKNOWN_ERROR
@@ -26,6 +25,7 @@
);
-my ($req_file, $req_id, $out_dir, $product, $mode, $dbname, $dbserver, $verbose, $save_temps);
-
+my ($no_update, $imagedb, $label);
+my ($req_file, $req_id, $outdir, $product, $mode, $dbname, $dbserver, $verbose, $save_temps);
+my ($job_id,$rownum); # stuff from the post-update world
#
# parse args
@@ -35,5 +35,8 @@
'file=s' => \$req_file,
'req_id=s' => \$req_id,
- 'out_dir=s' => \$out_dir,
+ 'job_id=s' => \$job_id,
+ 'rownum=s' => \$rownum,
+ 'outdir=s' => \$outdir,
+ 'label=s' => \$label,
'product=s' => \$product,
'mode=s' => \$mode,
@@ -44,25 +47,16 @@
) or pod2usage(2);
-my $err = "";
-
-if (!$req_file) {
- $err .= "--file is required\n";
-}
-if (!$req_id) {
- $err .= "--req_id is required\n";
-}
-if (!$out_dir) {
- $err .="--out_dir is required\n";
-}
-if (!$product) {
- $err .="--product is required\n";
-}
-
-
-die $err if ($err);
-
+die "invalid mode '$mode'" unless ($mode eq "list_uri") or ($mode eq "queue_job");
+die "--file or --job_id is required" if !$req_file;
+
+if ($mode ne "list_uri") {
+ die "req_id is required" if !$req_id;
+ die "outdir is required" if !$outdir;
+ die "product is required" if !$product;
+}
my $missing_tools;
my $pstamptool = can_run('pstamptool') or (warn "Can't find pstamptool" and $missing_tools =1);
-my $fakedresponse = can_run('fakedresponse.pl') or (warn "Can't find fakedresponse.pl" and $missing_tools =1);
+my $detectresponse = can_run('detectability_respond.pl') or
+ (warn "Can't find detectability_respond.pl" and $missing_tools = 1);
my $fields = can_run('fields') or (warn "Can't find fields" and $missing_tools =1);
@@ -77,6 +71,21 @@
}
+# just deal with these arguments once and for all
+$pstamptool .= " -dbname $dbname" if $dbname;
+$pstamptool .= " -dbserver $dbserver" if $dbserver;
+$detectresponse .= " --dbname $dbname" if $dbname;
+
+$no_update = 1 if $mode eq "list_job";
+
+# Unless we're running as a job, write the parse arguments in case we need to rerun this parsing.
+if (!$job_id) {
+ my $argslist = "$outdir/parse.args";
+ open ARGSLIST, ">$argslist" or my_die("failed to open argslist file $argslist", $job_id, $PS_EXIT_UNKNOWN_ERROR);
+ print ARGSLIST "--label $label --mode $mode --req_id $req_id --product $product --outdir $outdir --file $req_file\n";
+ close ARGSLIST;
+}
+
# get the query id and check the extname and version from the header
-my $fields_output;
+my $fields_output;
{
my $command = "echo $req_file | $fields -x 0 EXTNAME EXTVER QUERY_ID";
@@ -84,41 +93,29 @@
run(command => $command, verbose => $verbose);
-# fields doesn't return zero when it succeeds
-# unless ($success) {
-# print STDERR @$stderr_buf;
-# }
$fields_output = join "", @$stdout_buf;
}
my (undef, $extname, $extver, $req_name) = split " ", $fields_output;
-die "$req_file is missing one of EXTNAME EXTVER or QUERY_ID"
+my_die("$req_file is missing one of EXTNAME EXTVER or QUERY_ID", $PS_EXIT_PROG_ERROR)
if !(defined($extname) and defined($extver) and defined($req_name));
-
-die "$req_file has EXTNAME $extname not MOPS_DETECTABILITY_QUERY table"
- if $extname ne "MOPS_DETECTABILITY_QUERY";
-die "$req_file is version $extver expecting 1" if $extver ne 1;
-
-if (0) {
-$out_dir .= "/$req_name";
-}
-if (! -e $out_dir ) {
- mkdir $out_dir or die "cannot create output directory $out_dir";
-} elsif (! -d $out_dir ) {
- die "output fileset directory $out_dir exists but is not a directory";
-}
-
-#
-# XXX Eventually we will parse the file here, to look up the list of input images to be processed
-# and queue a job for each image.
-# The request file will be one of the arguments. Each job will look at all rows and create entries
-# in the response file the coordinates that overlap the image.
-#
-# In the meantime we don't parse the file here, pass it to the program fakedresponse.
-# It creates a response file with a line for each row in the request file
-
-my $response_file = "$out_dir/response.fits";
+my_die("$req_file has EXTNAME $extname not MOPS_DETECTABILITY_QUERY table", $PS_EXIT_PROG_ERROR)
+ if $extname ne "MOPS_DETECTABILITY_QUERY";
+my_die("$req_file is version $extver expecting 1", $PS_EXIT_PROG_ERROR)
+ if $extver ne 1;
+
+# Set up the workdir for this query.
+if (! -e $outdir ) {
+ mkdir $outdir or my_die("cannot create output directory $outdir", $PS_EXIT_PROG_ERROR);
+} elsif (! -d $outdir ) {
+ my_die ("output fileset directory $outdir exists but is not a directory", $PS_EXIT_PROG_ERROR);
+}
+
+
+# Pass along the request file to the response generator.
+my $response_file = "$outdir/${req_name}.dresponse.${req_id}.fits";
my $fault;
+my $data_to_update = '';
{
- my $command = "$fakedresponse --input $req_file --output $response_file --workdir $out_dir";
+ my $command = "$detectresponse --input $req_file --output $response_file --workdir $outdir";
$command .= " --save-temps" if $save_temps;
$command .= " --verbose" if $verbose;
@@ -127,26 +124,28 @@
run(command => $command, verbose => $verbose);
unless ($success) {
- warn("Unable to perform $command error code: $error_code");
- }
+ warn("Warn! Unable to perform $command error code: $error_code");
+ }
+ # Use the fault code to see if we can regenerate the missing data.
$fault = $error_code >> 8;
-}
-
-my $job_id;
+ if ($fault == $PSTAMP_NOT_AVAILABLE) {
+ $data_to_update = (split /\n/, (join "", @$stdout_buf))[-1];
+ }
+}
+
my $result;
-{
- my $command = "$pstamptool -addjob -req_id $req_id -outputBase $out_dir";
- $command .= " -job_type detect_query -state stop -fault $fault";
+# If we returned correctly with a valid response file, get a job ID
+# for the completed work, and move the response to a standardized name.
+if ($fault == 0) {
+ my $command = "$pstamptool -addjob -req_id $req_id -outputBase $outdir";
+ $command .= " -job_type detect_query -state stop -fault 0";
$command .= " -rownum 1";
- $command .= " -dbname $dbname" if $dbname;
- $command .= " -dbserver $dbserver" if $dbserver;
-
- my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
- run(command => $command, verbose => $verbose);
-
+
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
if ($success) {
$job_id = join "", @$stdout_buf;
chomp $job_id;
if ($job_id && -e $response_file) {
- rename $response_file, "$out_dir/response${job_id}.fits";
+ rename $response_file, "$outdir/response${job_id}.fits";
}
$result = 0;
@@ -156,18 +155,95 @@
}
}
-
-
+else {
+ # Failed to run correctly, which means that we need to queue a job and flag data for updating.
+ # Get the dependency id for the data we're requesting be updated.
+ my $dep_id = queue_update_run($req_id,$job_id,$outdir,$label,$data_to_update);
+
+ # Link this request to a job and link that job to any dependency
+ my $command = "$pstamptool -addjob -req_id $req_id -outputBase $outdir";
+ $command .= " -job_type detect_query -state run -fault 0";
+ $command .= " -rownum 1";
+ $command .= " -dep_id $dep_id" if $dep_id;
+
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+
+ if ($success) {
+ $job_id = join "", @$stdout_buf;
+ chomp $job_id;
+ if ($job_id && -e $response_file) {
+ # We shouldn't have a response file at this stage.
+ rename $response_file, "$outdir/response${job_id}.fits";
+ }
+ $result = 0;
+ } else {
+ warn("Unable to perform $command error code: $error_code");
+ $result = $error_code >> 8;
+ }
+}
+
+# This does not set the request state to stop. That will happen with the request_finish.pl script,
+# which will notice that we've inserted the stopped job and decide we're finished. Easy enough.
{
- my $command = "$pstamptool -updatereq -req_id $req_id -name $req_name -outProduct $product";
- $command .= " -fault $result" if $result;
- $command .= " -dbname $dbname" if $dbname;
- $command .= " -dbserver $dbserver" if $dbserver;
+ my $command = "$pstamptool -updatereq -req_id $req_id -set_name $req_name -set_outProduct $product";
+ $command .= " -set_fault $result" if $result;
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
run(command => $command, verbose => $verbose);
unless ($success) {
- die "$command failed";
+ my_die("$command failed",$PS_EXIT_UNKNOWN_ERROR);
}
}
exit 0;
+
+
+# If we have to queue an update run, do so and create a new dependent
+sub queue_update_run {
+ my ($req_id, $job_id, $outdir, $label, $data_to_update) = @_;
+
+ my ($state, $stage, $stage_id, $component, $need_magic, $imagedb) = split /\s+/, $data_to_update;
+
+ if (($state ne 'cleaned') and ($state ne 'update') and ($state ne 'goto_cleaned')) {
+ # We should have received one of these states, so if not, signal that we have a problem.
+ my_die("$stage $stage_id is in unexpected state $state", $PS_EXIT_PROG_ERROR);
+ }
+ my $dep_id;
+ my $command = "$pstamptool -getdependent -stage $stage -stage_id $stage_id -imagedb $imagedb -component $component ";
+ $command .= " -outdir $outdir";
+ $command .= " -need_magic" if $need_magic;
+
+ my $rlabel = "dq_ud_" . $label if $label;
+ $command .= " -rlabel $rlabel" if $rlabel;
+
+ if (!$no_update) {
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ my_die("$command failed", $PS_EXIT_UNKNOWN_ERROR);
+ }
+ my $output = join "", @$stdout_buf;
+ chomp $output;
+ $dep_id = $output;
+
+ my_die("pstamptool -getdependent returned invalid dep_id", $PS_EXIT_PROG_ERROR) if !$dep_id;
+ }
+ else {
+ print STDERR "skipping $command\n";
+ $dep_id = 42;
+ }
+
+ return($dep_id);
+}
+
+
+
+sub my_die {
+ my $msg = shift;
+ my $fault = shift;
+
+ carp $msg;
+
+ # we don't fault the request here pstamp_parser_run.pl handles that if necessary
+ exit $fault;
+}
Index: branches/pap/pstamp/scripts/pstamp_checkdependent.pl
===================================================================
--- branches/pap/pstamp/scripts/pstamp_checkdependent.pl (revision 27708)
+++ branches/pap/pstamp/scripts/pstamp_checkdependent.pl (revision 28003)
@@ -21,4 +21,9 @@
use Carp;
+# XXX: put this in a module somewhere
+my $IPP_DIFF_MODE_WARP_WARP = 1;
+my $IPP_DIFF_MODE_WARP_STACK = 2;
+my $IPP_DIFF_MODE_STACK_WARP = 3;
+my $IPP_DIFF_MODE_STACK_STACK = 4;
my ($dep_id, $stage, $stage_id, $component, $imagedb, $rlabel, $need_magic);
@@ -30,5 +35,5 @@
'stage_id=i' => \$stage_id,
'component=s' => \$component,
- 'imagedb=s' => \$imagedb, # dbname for images
+ 'imagedb=s' => \$imagedb, # dbname for images.
'rlabel=s' => \$rlabel,
'need_magic' => \$need_magic,
@@ -48,4 +53,5 @@
my $warptool = can_run('warptool') or (warn "Can't find warptool" and $missing_tools = 1);
my $difftool = can_run('difftool') or (warn "Can't find difftool" and $missing_tools = 1);
+my $stacktool = can_run('stacktool') or (warn "Can't find stacktool" and $missing_tools = 1);
my $magicdstool = can_run('magicdstool') or (warn "Can't find magicdstool" and $missing_tools = 1);
my $pstamptool = can_run('pstamptool') or (warn "Can't find pstamptool" and $missing_tools = 1);
@@ -55,9 +61,10 @@
}
+# Append imagedb to the ippTools
$chiptool .= " -dbname $imagedb";
$warptool .= " -dbname $imagedb";
$difftool .= " -dbname $imagedb";
+$stacktool .= " -dbname $imagedb";
$magicdstool .= " -dbname $imagedb";
-
my $ipprc = PS::IPP::Config->new();
@@ -66,5 +73,4 @@
$dbserver = metadataLookupStr($ipprc->{_siteConfig}, 'PS_DBSERVER');
}
-
@@ -86,5 +92,5 @@
} else {
if ($stage eq "chip") {
- $cmd = "$chiptool -processedimfile -chip_id $stage_id -class_id $component";
+ $cmd = "$chiptool -processedimfile -allfiles -chip_id $stage_id -class_id $component";
} elsif ($stage eq "warp") {
$cmd = "$warptool -warped -warp_id $stage_id -skycell_id $component";
@@ -96,17 +102,24 @@
}
-my $components = runToolAndParse($cmd, $verbose);
-
-my $n = scalar @$components;
-my_die("Unexpected number of components found $n", $PS_EXIT_PROG_ERROR) if ($n != 1);
+my $it = runToolAndParseExpectOne($cmd, $verbose);
+
+my_die("no components found", $PS_EXIT_PROG_ERROR) if ( !$it);
# Got "it"
-my $it = $components->[0];
-
+
+my $magic_ok = 0;
+if ($stage eq 'diff') {
+ if ($it->{diff_mode} == $IPP_DIFF_MODE_STACK_STACK) {
+ # stack stack diffs don't need magic, but since the warps need to have the chips destreaked
+ # in order to be processed se set need_magic in the database for all runs
+ # Now the diffs themselves don't need to be destreaked so
+ $magic_ok = 1;
+ }
+}
my $status = 0;
if (($it->{state} eq 'full') or ($it->{state} eq 'update') and ($whole_run or ($it->{data_state} eq 'full'))
- and (!$need_magic or $it->{magicked} > 0)) {
-
- # This Dependency is satisfied. all done
+ and (!$need_magic or $magic_ok or $it->{magicked} > 0)) {
+
+ # This Dependency is satisfied. All done!
my $command = "$pstamptool -updatedependent -set_state full -dep_id $dep_id";
$command .= " -dbname $dbname" if $dbname;
@@ -125,4 +138,6 @@
($whole_run or ($it->{data_state} eq 'cleaned'))) {
+ # Update has not been queued for this component yet.
+
# XXX: if $whole_run this is going to repeatedly call -setimfiletoupdate. This will be a no-op
# but is wasteful.
@@ -130,6 +145,5 @@
# need to queue the updates
if ($stage eq 'chip') {
- # queue_update_chip takes an array so that queue_update_warp can pass a set of
- # chips to it
+ # queue_update_chip takes an array so that queue_update_warp can pass it set of chips
my $chips = [$it];
$status = queue_update_chip($it->{chip_id}, $whole_run, $chips, $rlabel, $need_magic);
@@ -142,11 +156,30 @@
}
} else {
+ # update for this component has been queued by hasn't completed yet or the state is an error state
+ # XXX: in this state we aren't checking the dependents anymore. If something goes wrong this
+ # will never complete. Consider calling the 'queue_update' functions again just to check. They are relatively cheap
+
+
print "${stage}Run $stage_id state is $it->{state} $component data_state is $it->{data_state}\n";
# detect states that cannot be updated and update the job state
- # XXX: we should be more assertive here and check for the specific states where we can continue
+ # XXX: Perhaps I should be more assertive here and check for the specific states that we know
+ # that we can continue.
+
my $state = $it->{state};
my $job_fault = 0;
- if (($state =~ /scrub/) or ($state =~ /purge/)) {
+
+ # temporary hack to deal with data with old burntool tables. This check is now done by pstampparse.pl
+ # so runs in this state should not get queued
+ if ($stage eq 'chip') {
+ my $burntool_state = $it->{burntool_state};
+ if ($burntool_state and (abs($burntool_state) < 14)) {
+ print STDERR "chip $it->{chip_id} $it->{class_id} has burntool_state $burntool_state. Not avaiable.\n";
+ $job_fault = $PSTAMP_NOT_AVAILABLE;
+ }
+ }
+ if ($state eq 'error_cleaned') {
+ $job_fault = $PSTAMP_NOT_AVAILABLE;
+ } elsif (($state =~ /scrub/) or ($state =~ /purge/)) {
# jobs must have changed state since depenency was made
print STDERR "Dependency cannot be satisfied\n";
@@ -156,4 +189,8 @@
my_die ("Unexpected state for ${stage}Run $stage_id $state", $PS_EXIT_PROG_ERROR);
}
+ if (!$job_fault and ($stage eq 'chip')) {
+ # chip processing is done, start destreaking.
+ $job_fault = queue_update_magicDSRun($stage, $stage_id, $rlabel, $need_magic, $it->{raw_magicked}, $it->{dsRun_state});
+ }
if ($job_fault) {
faultJobs($state, $stage, $stage_id, $job_fault);
@@ -171,4 +208,5 @@
my $need_magic = shift;
+
my $dsRun_state;
my $raw_all_magicked = 1; # this gets cleared if any of the inputs aren't destreaked
@@ -178,5 +216,7 @@
$raw_all_magicked &= ($chip->{raw_magicked} > 0);
- if ($chip->{data_state} ne 'update') {
+ if ($chip->{state} =~ /error/) {
+ faultJobs('stop', undef, undef, $PSTAMP_GONE);
+ } elsif (($chip->{data_state} ne 'update') and ($chip->{data_state} ne 'full')) {
my $command = "$chiptool -setimfiletoupdate -chip_id $chip_id -class_id $chip->{class_id}";
$command .= " -set_label $rlabel" if $rlabel;
@@ -213,7 +253,364 @@
}
+ my $status = queue_update_magicDSRun('chip', $chip_id, $rlabel, $need_magic, $raw_all_magicked, $dsRun_state);
+
+ return $status;
+}
+
+sub queue_update_warp {
+ # check status of input chips. If state is not updatable set error code for job
+
+ # if chipProcessedImfile.state is cleaned call queue_update_chip
+
+ # need to code warptool -setskyfiletoupdate
+ my $metadata = shift;
+ my $whole_run = shift; # if true queue entire run for update
+ my $rlabel = shift; # if defined a new label for the chipRun
+ my $need_magic = shift;
+
+ my $raw_all_magicked = 1; # this gets cleared if any of the inputs aren't destreaked
+
+ my $warp_id = $metadata->{warp_id};
+ my $state = $metadata->{state};
+ if ($state =~ /error/) {
+ print STDERR "warpRun $warp_id has state $state faulting jobs\n";
+ faultJobs('stop', undef, undef, $PSTAMP_GONE);
+ exit 0;
+ }
+
+ if (!$whole_run) {
+ my $skycell = $metadata;
+ my $skycell_id = $skycell->{skycell_id};
+
+ my $command = "$warptool -scmap -warp_id $warp_id -skycell_id $skycell_id";
+ my $data = runToolAndParse($command, $verbose);
+ my_die("failed to find warpSkyCelllMap for warpRun $warp_id skycell_id $skycell_id", $PS_EXIT_UNKNOWN_ERROR)
+ if !$data or scalar @$data == 0;
+
+ my $chips_ready = 1;
+ my @chipsToUpdate;
+ my $chip_id;
+ foreach my $chip (@$data) {
+ $chip_id = $chip->{chip_id};
+ if (($chip->{data_state} ne 'full') or ($need_magic and ($chip->{magicked} <= 0))) {
+ $chips_ready = 0;
+ push @chipsToUpdate, $chip;
+ } else {
+ # this chip is good to go
+ }
+ }
+
+ if ($chips_ready) {
+ # the reason we defer setting the warp to update is so that we can handle error conditions at previous
+ # stages more easily.
+ my $command = "$warptool -setskyfiletoupdate -warp_id $warp_id -skycell_id $skycell->{skycell_id}";
+ $command .= " -set_label $rlabel" if $rlabel;
+
+ if (!$no_update) {
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ my_die("failed to queue ${stage}Run $stage_id $component for update", $PS_EXIT_UNKNOWN_ERROR);
+ }
+ } else {
+ print "skipping $command\n";
+ }
+ } elsif (scalar @chipsToUpdate > 0) {
+ return queue_update_chip($chip_id, 0, \@chipsToUpdate, $rlabel, $need_magic);
+ }
+ } else {
+ my $warpRun = $metadata;
+ my $command = "$chiptool -listrun -chip_id $warpRun->{chip_id}";
+ my $data = runToolAndParse($command, $verbose);
+ my_die("failed to find chipRun $warpRun->{chip_id} for warpRun $warp_id", $PS_EXIT_UNKNOWN_ERROR)
+ if !$data or scalar @$data != 1;
+
+ my $chipRun = $data->[0];
+
+ my $chipRunState = $chipRun->{state};
+ if (($chipRunState =~ /purge/) or ($chipRunState =~ /scrub/)) {
+ print STDERR "warpRun $warp_id depends on chipRun $chipRun->{chip_id} which is in state $chipRunState\n";
+ faultJobs('stop', 'warp', $warp_id, $PSTAMP_GONE);
+ return 0;
+ }
+ my $warpRunState = $warpRun->{state};
+ if (($chipRunState eq 'full') and (! $need_magic or ($chipRun->{magicked} > 0)) and ($warpRunState eq 'cleaned')) {
+ # The inputs and outputs are ready. Queue the warpRun for update.
+
+ # providing no -skycell_id arguments changes all skyfiles with data_state = 'cleaned' to 'update'
+ my $command = "$warptool -setskyfiletoupdate -warp_id $warp_id";
+ $command .= " -set_label $rlabel" if $rlabel;
+
+ if (!$no_update) {
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ my_die("failed to queue ${stage}Run $stage_id $component for update", $PS_EXIT_UNKNOWN_ERROR);
+ }
+ } else {
+ print "skipping $command\n";
+ }
+ } elsif ($chipRunState eq 'cleaned' or
+ (($chipRun->{state} eq 'full') and ($need_magic and ($chipRun->{magicked} < 0) and ($chipRun->{dsRun_state} ne 'new')))) {
+ my $data = [$chipRun];
+ return queue_update_chip($chipRun->{chip_id}, 1, $data, $rlabel, $need_magic);
+ }
+ }
+
+ # return value may be the return status of script so zero is good
+ return 0;
+}
+sub queue_update_diff {
+ my $metadata = shift;
+ my $whole_run = shift; # if true queue entire run for update
+ my $rlabel = shift; # if defined a new label for the chipRun
+ my $need_magic = shift;
+
+ my $diff_id = $metadata->{diff_id};
+ my $diff_mode = $metadata->{diff_mode};
+ if (!$whole_run) {
+ my $skycell = $metadata;
+ my $skycell_id = $skycell->{skycell_id};
+
+ if ($diff_mode == $IPP_DIFF_MODE_WARP_STACK ) {
+ # check the state of the template stack
+ my $command = "$stacktool -sumskyfile -stack_id $skycell->{stack2}";
+ my $stack = runToolAndParseExpectOne($command, $verbose);
+ my_die("failed to find stackSumSkyfile for stack_id $skycell->{stack2}", $PS_EXIT_UNKNOWN_ERROR) if !$stack;
+
+ if ($stack->{state} ne 'full') {
+ print STDERR "template stack for diffRun $diff_id $skycell_id is not in full state faulting jobs\n";
+ # this faults all jobs depending on this dep_id
+ faultJobs('stop', 'diff', $diff_id, $PSTAMP_GONE);
+ return $PSTAMP_GONE;
+ }
+
+ # now check the warp
+ $command = "$warptool -warped -warp_id $skycell->{warp1} -skycell_id $skycell_id";
+ my $warp = runToolAndParseExpectOne($command, $verbose);
+ my_die("failed to find warpSkyfile for warpRun $skycell->{warp1} skycell_id $skycell_id", $PS_EXIT_UNKNOWN_ERROR) if !$warp;
+
+ if ($warp->{data_state} ne 'full') {
+ return queue_update_warp($warp, 0, $rlabel, $need_magic);
+ }
+ # warps are ready fall through and queue the diff update
+ } elsif ($diff_mode eq $IPP_DIFF_MODE_WARP_WARP) {
+ my $command = "$warptool -warped -warp_id $skycell->{warp1} -skycell_id $skycell_id";
+ my $warp1 = runToolAndParseExpectOne($command, $verbose);
+ my_die("failed to find warpSkyfile for warpRun $skycell->{warp1} skycell_id $skycell_id", $PS_EXIT_UNKNOWN_ERROR) if !$warp1;
+
+ my $warps_ready = 1;
+ my $status = 0;
+ if ($warp1->{data_state} ne 'full') {
+ $warps_ready = 0;
+ $status = queue_update_warp($warp1, 0, $rlabel, $need_magic);
+ if ($status) {
+ return $status;
+ }
+ }
+ $command = "$warptool -warped -warp_id $skycell->{warp2} -skycell_id $skycell_id";
+ my $warp2 = runToolAndParseExpectOne($command, $verbose);
+ my_die("failed to find warpSkyfile for warpRun $skycell->{warp2} skycell_id $skycell_id", $PS_EXIT_UNKNOWN_ERROR) if !$warp2;
+
+ if ($warp2->{data_state} ne 'full') {
+ $warps_ready = 0;
+ $status = queue_update_warp($warp2, 0, $rlabel, $need_magic);
+ }
+
+ if (!$warps_ready) {
+ # don't queue the diff update yet
+ return $status;
+ }
+
+ } elsif ($diff_mode == $IPP_DIFF_MODE_STACK_STACK ) {
+ # check the state of the input stack
+ my $command = "$stacktool -sumskyfile -stack_id $skycell->{stack2}";
+ my $stack1 = runToolAndParseExpectOne($command, $verbose);
+ my_die("failed to find stackSumSkyfile for stack_id $skycell->{stack2}", $PS_EXIT_UNKNOWN_ERROR) if !$stack1;
+
+ if ($stack1->{state} ne 'full') {
+ print STDERR "input stack for diffRun $diff_id $skycell_id is not in full state faulting jobs\n";
+ # this faults all jobs depending on this dep_id
+ faultJobs('stop', 'diff', $diff_id, $PSTAMP_GONE);
+ return $PSTAMP_GONE;
+ }
+ # check the state of the template stack
+ $command = "$stacktool -sumskyfile -stack_id $skycell->{stack2}";
+ my $stack2 = runToolAndParseExpectOne($command, $verbose);
+ my_die("failed to find stackSumSkyfile for stack_id $skycell->{stack2}", $PS_EXIT_UNKNOWN_ERROR) if !$stack2;
+
+ if ($stack2->{state} ne 'full') {
+ print STDERR "template stack for diffRun $diff_id $skycell_id is not in full state faulting jobs\n";
+ # this faults all jobs depending on this dep_id
+ faultJobs('stop', 'diff', $diff_id, $PSTAMP_GONE);
+ return $PSTAMP_GONE;
+ }
+
+ # inputs are ready fall through and queue the diff update
+ } elsif ($diff_mode == $IPP_DIFF_MODE_STACK_WARP ) {
+ # check the state of the input stack
+ my $command = "$stacktool -sumskyfile -stack_id $skycell->{stack1}";
+ my $stack = runToolAndParseExpectOne($command, $verbose);
+ my_die("failed to find stackSumSkyfile for stack_id $skycell->{stack1}", $PS_EXIT_UNKNOWN_ERROR) if !$stack;
+
+ if ($stack->{state} ne 'full') {
+ print STDERR "input stack for diffRun $diff_id $skycell_id is not in full state faulting jobs\n";
+ # this faults all jobs depending on this dep_id
+ faultJobs('stop', 'diff', $diff_id, $PSTAMP_GONE);
+ return $PSTAMP_GONE;
+ }
+
+ # now check the template warp
+ $command = "$warptool -warped -warp_id $skycell->{warp2} -skycell_id $skycell_id";
+ my $warp = runToolAndParseExpectOne($command, $verbose);
+ my_die("failed to find warpSkyfile for warpRun $skycell->{warp2} skycell_id $skycell_id", $PS_EXIT_UNKNOWN_ERROR) if !$warp;
+
+ if ($warp->{data_state} ne 'full') {
+ return queue_update_warp($warp, 0, $rlabel, $need_magic);
+ }
+ # warps are ready fall through and queue the diff update
+ } else {
+ my_die("unexpected diff_mode found: $diff_mode", $PS_EXIT_PROG_ERROR);
+ }
+
+ my $command = "$difftool -setskyfiletoupdate -diff_id $diff_id -skycell_id $skycell_id";
+ $command .= " -set_label $rlabel" if $rlabel;
+
+ if (!$no_update) {
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ my_die("failed to queue ${stage}Run $stage_id $component for update", $PS_EXIT_UNKNOWN_ERROR);
+ }
+ } else {
+ print "skipping $command\n";
+ }
+ } else {
+ # XXX todo whole_run
+ return $PS_EXIT_PROG_ERROR;
+ }
+
+ # return value is a unix style exit status so zero is good
+ return 0;
+}
+
+
+# run a command that produces metadata output and parse the results into an array of objects
+sub runToolAndParse {
+ my $command = shift;
+ my $verbose = shift;
+
+ my ($program) = split " ", $command;
+ $program = basename($program);
+
+ print "Running $command\n" if !$verbose;
+ my $start_tool = DateTime->now->mjd;
+ # run the command and parse the output
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ print STDERR @$stderr_buf if !$verbose;
+ return undef;
+ }
+
+ my $now = DateTime->now->mjd;
+ my $dtime_tool = (DateTime->now->mjd - $start_tool) * 86400.;
+ print "Time to run $program: $dtime_tool\n";
+
+ my $buf = join "", @$stdout_buf;
+ if (!$buf) {
+ return undef;
+ }
+
+ my $start_parse = DateTime->now->mjd;
+
+ my $mdcParser = PS::IPP::Metadata::Config->new;
+ my $results = parse_md_fast($mdcParser, $buf)
+ or my_die ("Unable to parse metadata config doc", $PS_EXIT_UNKNOWN_ERROR);
+
+ my $dtime_parse = (DateTime->now->mjd - $start_parse) * 86400.;
+ print "Time to parse results from $program: $dtime_parse\n";
+
+ return $results;
+}
+
+# run an command returning metadata where we expect 1 entry
+sub runToolAndParseExpectOne {
+ my $command = shift;
+ my $verbose = shift;
+
+ my $data = runToolAndParse($command, $verbose);
+
+ if (!$data) {
+ return undef;
+ }
+
+ my $n = scalar @$data;
+ if ($n > 1) {
+ my_die("Unexpected number of components $n returned by $command", $PS_EXIT_PROG_ERROR);
+ }
+
+ return $data->[0];
+}
+
+# splits meta data config input stream into single units to work around the pathalogically
+# slow parser. This is similar to and adapted from code in various ippScripts.
+sub parse_md_fast {
+ my $mdcParser = shift;
+ my $input = shift;
+ my $output = ();
+
+ my @whole = split /\n/, $input;
+ my @single = ();
+
+ my $n;
+ while ( ($n = @whole) > 0) {
+ my $value = shift @whole;
+ push @single, $value;
+ if ($value =~ /^\s*END\s*$/) {
+ push @single, "\n";
+
+ my $list = parse_md_list( $mdcParser->parse( join("\n", @single ) ) ) or
+ print STDERR "Unable to parse metdata config doc" and return undef;
+ push @$output, $list->[0];
+
+ @single = ();
+ }
+ }
+ return $output;
+}
+
+sub faultJobs {
+ my ($state, $stage, $stage_id, $job_fault) = @_;
+
+ my $command = "$pstamptool -updatejob -set_state stop -set_fault $job_fault -dep_id $dep_id";
+ $command .= " -dbname $dbname" if $dbname;
+ $command .= " -dbserver $dbserver" if $dbserver;
+ if (!$no_update) {
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ my_die("failed to set pstampJob.fault for dep_id: $dep_id",
+ $PS_EXIT_UNKNOWN_ERROR);
+ }
+ } else {
+ print "skipping $command\n";
+ }
+}
+
+sub queue_update_magicDSRun {
+ my $stage = shift;
+ my $stage_id = shift;
+ my $rlabel = shift;
+ my $need_magic = shift;
+ my $input_magicked = shift;
+ my $dsRun_state = shift;
+
+ # XXX: this code assumes that destreaking is handled at the chip stage
+ my_die ("queue_update_magicDSRun only works for stage chip", $PS_EXIT_PROG_ERROR) if $stage ne 'chip';
+
# if called from queue_update_warp dsRun_state is unknown. Go find it.
if (!$dsRun_state) {
- my $command = "$chiptool -listrun -chip_id $chip_id";
+ my $command = "$chiptool -listrun -chip_id $stage_id";
my $data = runToolAndParse($command, $verbose);
my $chipRun = $data->[0];
@@ -222,5 +619,5 @@
# if the input file is already magicked no need to queue destreaking for this chipRun
- if ($need_magic and !$raw_all_magicked) {
+ if ($need_magic and !$input_magicked) {
if ($dsRun_state eq 'cleaned') {
my $command = "$magicdstool -updaterun -set_state new -stage $stage -stage_id $stage_id";
@@ -236,4 +633,7 @@
print "skipping $command\n";
}
+ } elsif ($dsRun_state eq 'failed_revert') {
+ print "magicDSRun.state = $dsRun_state for chipRun $stage_id is in state failed_revert cannot update";
+ faultJobs('stop', undef, undef, $PSTAMP_NOT_AVAILABLE);
} else {
print "magicDSRun.state = $dsRun_state for chipRun $stage_id";
@@ -242,177 +642,14 @@
}
}
-
- # return value may be the return status of script so zero is good
return 0;
}
-sub queue_update_warp {
- # check status of input chips. If state is not updatable set error code for job
-
- # if chipProcessedImfile.state is cleaned call queue_update_chip
-
- # need to code warptool -setskyfiletoupdate
- my $metadata = shift;
- my $whole_run = shift; # if true queue entire run for update
- my $rlabel = shift; # if defined a new label for the chipRun
- my $need_magic = shift;
-
- my $raw_all_magicked = 1; # this gets cleared if any of the inputs aren't destreaked
-
- my $warp_id = $metadata->{warp_id};
- if (!$whole_run) {
- my $skycell = $metadata;
- my $skycell_id = $skycell->{skycell_id};
-
- my $command = "$warptool -scmap -warp_id $warp_id -skycell_id $skycell_id";
- my $data = runToolAndParse($command, $verbose);
- my_die("failed to find warpSkyCelllMap for warpRun $warp_id skycell_id $skycell_id", $PS_EXIT_UNKNOWN_ERROR)
- if !$data or scalar @$data == 0;
-
- my $good_to_go = 1;
- my @chipsToUpdate;
- my $chip_id;
- foreach my $chip (@$data) {
- $chip_id = $chip->{chip_id};
- if (($chip->{data_state} ne 'full') or ($need_magic and ($chip->{magicked} <= 0))) {
- $good_to_go = 0;
- push @chipsToUpdate, $chip;
- } else {
- # this chip is good to go
- }
- }
-
- if ($good_to_go) {
- my $command = "$warptool -setskyfiletoupdate -warp_id $warp_id -skycell_id $skycell->{skycell_id}";
- $command .= " -set_label $rlabel" if $rlabel;
-
- if (!$no_update) {
- my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
- run(command => $command, verbose => $verbose);
- unless ($success) {
- my_die("failed to queue ${stage}Run $stage_id $component for update", $PS_EXIT_UNKNOWN_ERROR);
- }
- } else {
- print "skipping $command\n";
- }
- } elsif (scalar @chipsToUpdate > 0) {
- return queue_update_chip($chip_id, 0, \@chipsToUpdate, $rlabel, $need_magic);
- }
- } else {
- my $warpRun = $metadata;
- my $command = "$chiptool -listrun -chip_id $warpRun->{chip_id}";
- my $data = runToolAndParse($command, $verbose);
- my_die("failed to find chipRun $warpRun->{chip_id} for warpRun $warp_id", $PS_EXIT_UNKNOWN_ERROR)
- if !$data or scalar @$data != 1;
-
- my $chipRun = $data->[0];
-
- my $chipRunState = $chipRun->{state};
- if (($chipRunState =~ /purge/) or ($chipRunState =~ /scrub/)) {
- print STDERR "warpRun $warp_id depends on chipRun $chipRun->{chip_id} which is in state $chipRunState\n";
- faultJobs('stop', 'warp', $warp_id, $PSTAMP_GONE);
- return 0;
- }
- my $warpRunState = $warpRun->{state};
- if (($chipRunState eq 'full') and (! $need_magic or ($chipRun->{magicked} > 0)) and ($warpRunState eq 'cleaned')) {
- # The inputs and outputs are ready. Queue the warpRun for update.
-
- # providing no -skycell_id arguments changes all skyfiles with data_state = 'cleaned' to 'update'
- my $command = "$warptool -setskyfiletoupdate -warp_id $warp_id";
- $command .= " -set_label $rlabel" if $rlabel;
-
- if (!$no_update) {
- my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
- run(command => $command, verbose => $verbose);
- unless ($success) {
- my_die("failed to queue ${stage}Run $stage_id $component for update", $PS_EXIT_UNKNOWN_ERROR);
- }
- } else {
- print "skipping $command\n";
- }
- } elsif ($chipRunState eq 'cleaned' or
- (($chipRun->{state} eq 'full') and ($need_magic and ($chipRun->{magicked} < 0) and ($chipRun->{dsRun_state} ne 'new')))) {
- my $data = [$chipRun];
- return queue_update_chip($chipRun->{chip_id}, 1, $data, $rlabel, $need_magic);
- }
- }
-
- # return value may be the return status of script so zero is good
- return 0;
-}
-sub queue_update_diff {
- return 0;
-}
-
-
-# run a command that produces metadata output and parse the results into an array of objects
-sub runToolAndParse {
- my $command = shift;
- my $verbose = shift;
-
- my ($program) = split " ", $command;
- $program = basename($program);
-
- print "Running $command\n" if !$verbose;
- my $start_tool = DateTime->now->mjd;
- # run the command and parse the output
- my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
- run(command => $command, verbose => $verbose);
- unless ($success) {
- print STDERR @$stderr_buf if !$verbose;
- return undef;
- }
-
- my $now = DateTime->now->mjd;
- my $dtime_tool = (DateTime->now->mjd - $start_tool) * 86400.;
- print "Time to run $program: $dtime_tool\n";
-
- my $buf = join "", @$stdout_buf;
- if (!$buf) {
- return undef;
- }
-
- my $start_parse = DateTime->now->mjd;
-
- my $mdcParser = PS::IPP::Metadata::Config->new;
- my $results = parse_md_fast($mdcParser, $buf)
- or my_die ("Unable to parse metadata config doc", $PS_EXIT_UNKNOWN_ERROR);
-
- my $dtime_parse = (DateTime->now->mjd - $start_parse) * 86400.;
- print "Time to parse results from $program: $dtime_parse\n";
-
- return $results;
-}
-
-# splits meta data config input stream into single units to work around the pathalogically
-# slow parser. This is similar to and adapted from code in various ippScripts.
-sub parse_md_fast {
- my $mdcParser = shift;
- my $input = shift;
- my $output = ();
-
- my @whole = split /\n/, $input;
- my @single = ();
-
- my $n;
- while ( ($n = @whole) > 0) {
- my $value = shift @whole;
- push @single, $value;
- if ($value =~ /^\s*END\s*$/) {
- push @single, "\n";
-
- my $list = parse_md_list( $mdcParser->parse( join("\n", @single ) ) ) or
- print STDERR "Unable to parse metdata config doc" and return undef;
- push @$output, $list->[0];
-
- @single = ();
- }
- }
- return $output;
-}
-sub faultJobs {
- my ($state, $stage, $stage_id, $job_fault) = @_;
-
- my $command = "$pstamptool -updatejob -state stop -fault $job_fault -dep_id $dep_id";
+sub my_die
+{
+ my $msg = shift;
+ my $fault = shift;
+ carp $msg;
+
+ my $command = "$pstamptool -updatedependent -set_fault $fault -dep_id $dep_id";
$command .= " -dbname $dbname" if $dbname;
$command .= " -dbserver $dbserver" if $dbserver;
@@ -421,19 +658,10 @@
run(command => $command, verbose => $verbose);
unless ($success) {
- my_die("failed to set pstamJob.fault for dep_id: $dep_id",
- $PS_EXIT_UNKNOWN_ERROR);
+ carp "$cmd failed";
}
} else {
print "skipping $command\n";
}
-}
-
-sub my_die
-{
- my $msg = shift;
- my $fault = shift;
-
- carp $msg;
-
- return $fault;
-}
+
+ exit $fault;
+}
Index: branches/pap/pstamp/scripts/pstamp_cleanup.pl
===================================================================
--- branches/pap/pstamp/scripts/pstamp_cleanup.pl (revision 28003)
+++ branches/pap/pstamp/scripts/pstamp_cleanup.pl (revision 28003)
@@ -0,0 +1,201 @@
+#!/bin/env perl
+###
+### pstamp_cleanup.pl
+### delete all data associated with a postage stamp request
+###
+### This script should probably be called request_cleanup.pl since it handles more than postage stamp requests
+###
+
+use warnings;
+use strict;
+
+use Sys::Hostname;
+use Getopt::Long qw( GetOptions );
+use File::Basename qw( basename dirname);
+use File::Copy;
+use POSIX qw( strftime );
+use Carp;
+use IPC::Cmd 0.36 qw( can_run run );
+
+use PS::IPP::Metadata::Config;
+use PS::IPP::Metadata::Stats;
+use PS::IPP::Metadata::List qw( parse_md_list );
+
+use PS::IPP::Config qw( :standard );
+
+my $req_id;
+my $uri;
+my $name;
+my $outdir;
+my $reqType;
+my $redirect_output;
+my $product;
+my $label;
+my $verbose;
+my $dbname;
+my $dbserver;
+my $no_update;
+
+GetOptions(
+ 'req_id=s' => \$req_id,
+ 'name=s' => \$name,
+ 'outdir=s' => \$outdir,
+ 'reqType=s' => \$reqType,
+ 'product=s' => \$product,
+ 'uri=s' => \$uri,
+ 'outdir=s' => \$outdir,
+ 'redirect-output' => \$redirect_output,
+ 'verbose' => \$verbose,
+ 'dbname=s' => \$dbname,
+ 'dbserver=s' => \$dbserver,
+ 'no-update' => \$no_update,
+);
+
+if ($verbose) {
+ my $host = hostname();
+ print "\n\n";
+ print "Starting script $0 on $host\n\n";
+}
+
+my $missing_tools;
+
+my $pstamptool = can_run('pstamptool') or (warn "Can't find pstamptool" and $missing_tools = 1);
+my $dsreg = can_run('dsreg') or (warn "Can't find dsreg" and $missing_tools = 1);
+
+if ($missing_tools) {
+ warn("Can't find required tools.");
+ exit ($PS_EXIT_CONFIG_ERROR);
+}
+
+
+die("--req_id --name --product and --outdir are required") if !defined($req_id) or !defined($product) or !defined($name) or !defined $outdir;
+
+my $ipprc = PS::IPP::Config->new(); # IPP Configuration
+
+my $pstamp_workdir = metadataLookupStr($ipprc->{_siteConfig}, 'PSTAMP_WORKDIR');
+exit ($PS_EXIT_CONFIG_ERROR) unless defined $pstamp_workdir; # lookup failure outputs a message
+
+if (!$dbserver) {
+ $dbserver = metadataLookupStr($ipprc->{_siteConfig}, 'PS_DBSERVER');
+}
+
+my_die("Cleanup not yet supported for reqType: $reqType", $req_id, $PS_EXIT_UNKNOWN_ERROR)
+ if ($reqType ne "pstamp") and ($reqType ne "NULL");
+
+my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files
+
+if ($product ne 'NULL' and $name ne 'NULL') {
+ my $command = "$dsreg --product $product --del $name --rm --force";
+
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ my_die("Unable to perform $command error code: $error_code", $req_id, $PS_EXIT_UNKNOWN_ERROR);
+ }
+}
+
+# now go find the workdir for this request
+# XXX: we finally *have* to store this in the database
+$outdir = undef if $outdir eq "NULL";
+
+if ($outdir) {
+ delete_workdir($outdir);
+} else {
+ if ($uri ne 'NULL') {
+ # use the URI to find the workdir
+ $outdir = dirname($uri);
+ if (index($outdir, $pstamp_workdir) == 0) {
+ print "pstamp workdir for req_id $req_id is $outdir\n";
+ delete_workdir($outdir);
+ } else {
+ print "no outdir found for $req_id\n";
+ }
+ }
+ if (!$outdir) {
+ # gotta go look for it in the subdirectories of $pstamp_workdir
+ my $command = "find $pstamp_workdir -maxdepth 2 -type d -name $req_id";
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ my $rc = $error_code >> 8;
+ my_die("Unable to perform $command error code: $error_code", $req_id, $rc);
+ }
+ if ($stdout_buf) {
+ # for historical reasons (a bug) there may be more than one outdir
+ my @outdirs = split "\n", join "", $stdout_buf;
+ foreach $outdir (@outdirs) {
+ chomp $outdir;
+ delete_workdir($outdir);
+ }
+ } else {
+ print "no outdir found for $req_id\n";
+ }
+ }
+}
+
+
+{
+ my $command = "$pstamptool -updatereq -req_id $req_id -set_state cleaned";
+ $command .= " -dbname $dbname" if $dbname;
+ $command .= " -dbserver $dbserver" if $dbserver;
+ unless ($no_update) {
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ die("Unable to perform $command error code: $error_code");
+ }
+ } else {
+ print "skipping $command\n";
+ }
+}
+
+exit 0;
+
+
+sub delete_workdir {
+ my $dir = shift;
+
+ if (!-e $dir) {
+ print "outdir $dir does not exist\n";
+ return 0;
+ }
+ if (!-d $dir) {
+ print "outdir $dir is not a directory\n";
+ return 0;
+ }
+
+ my $command = "rm -r $dir";
+ unless ($no_update) {
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ my $rc = $error_code >> 8;
+ my_die("Unable to perform $command return status: $rc", $req_id, $rc);
+ }
+ } else {
+ print "skipping $command\n";
+ }
+ return 0;
+}
+
+sub my_die {
+ my $msg = shift;
+ my $req_id = shift;
+ my $fault = shift;
+
+ carp($msg);
+
+ my $command = "$pstamptool -updatereq -req_id $req_id -set_fault $fault";
+ $command .= " -dbname $dbname" if $dbname;
+ $command .= " -dbserver $dbserver" if $dbserver;
+ unless ($no_update) {
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ die("Unable to perform $command error code: $error_code");
+ }
+ } else {
+ print "skipping $command\n";
+ }
+ exit $fault;
+}
Index: branches/pap/pstamp/scripts/pstamp_dorequest.pl
===================================================================
--- branches/pap/pstamp/scripts/pstamp_dorequest.pl (revision 27708)
+++ branches/pap/pstamp/scripts/pstamp_dorequest.pl (revision 28003)
@@ -106,5 +106,5 @@
{
## TODO: what about request status
- my $command = "$pstamptool -updatereq -req_id $request_id -state stop";
+ my $command = "$pstamptool -updatereq -req_id $request_id -set_state stop";
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
run(command => $command, verbose => $verbosity);
Index: branches/pap/pstamp/scripts/pstamp_finish.pl
===================================================================
--- branches/pap/pstamp/scripts/pstamp_finish.pl (revision 27708)
+++ branches/pap/pstamp/scripts/pstamp_finish.pl (revision 28003)
@@ -25,5 +25,7 @@
use PS::IPP::PStamp::Job qw( :standard );
-my ( $req_id, $req_name, $req_file, $out_dir, $product, $dbname, $dbserver, $verbose, $save_temps, $redirect_output);
+my %imagedb_cache;
+
+my ( $req_id, $req_name, $req_file, $outdir, $product, $dbname, $dbserver, $verbose, $save_temps, $redirect_output);
# the char to the right of the bar may be used as a single - alias for the longer name
@@ -34,5 +36,5 @@
'req_file=s' => \$req_file,
'product=s' => \$product,
- 'out_dir=s' => \$out_dir,
+ 'outdir=s' => \$outdir,
'dbname=s' => \$dbname,
'dbserver=s' => \$dbserver,
@@ -44,10 +46,13 @@
pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV;
-die "usage: --req_id id --req_name name --req_file file --product product --out_dir output_directory [--dbname dbname --verbose]\n"
- if !$req_id or !$req_name or !$req_file or !$product or !$out_dir;
+die "usage: --req_id id --req_name name --req_file file --product product --outdir output_directory [--dbname dbname --verbose]\n"
+ if !$req_id or !$req_name or !$req_file or !$product or !$outdir;
+
+die "outdir is NULL\n" if $outdir eq "NULL";
my $ipprc = PS::IPP::Config->new(); # IPP Configuration
if ($redirect_output) {
- my $logDest = "$out_dir/psfinish.$req_id.log";
+ # XXX: what happens here if the directory does not exist? We check below
+ my $logDest = "$outdir/psfinish.$req_id.log";
$ipprc->redirect_output($logDest);
}
@@ -70,9 +75,7 @@
if ($product eq "NULL") {
+ # nothing more to do
stop_request_and_exit($req_id, $PS_EXIT_PROG_ERROR);
}
-
-my $outputDataStoreRoot = metadataLookupStr($ipprc->{_siteConfig}, 'DATA_STORE_ROOT');
-exit ($PS_EXIT_CONFIG_ERROR) unless defined $outputDataStoreRoot; # lookup failure outputs a message
my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files
@@ -82,20 +85,20 @@
my $fileset = $req_name;
- print STDERR "product: $product REQ_NAME: $req_name $out_dir\n" if $verbose;
-
- if (!-e $out_dir) {
+ print STDERR "product: $product REQ_NAME: $req_name $outdir\n" if $verbose;
+
+ if (!-e $outdir) {
# something must have gone wrong parsing the request
- print STDERR "output directory $out_dir does not exist\n";
-
- if (!mkdir $out_dir) {
- print STDERR "cannot create output directory $out_dir";
+ print STDERR "output directory $outdir does not exist\n";
+
+ if (!mkdir $outdir) {
+ print STDERR "cannot create output directory $outdir";
stop_request_and_exit($req_id, $PS_EXIT_UNKNOWN_ERROR);
}
- } elsif (! -d $out_dir ) {
+ } elsif (! -d $outdir ) {
# XXX TODO: fault the request so we pstamp_finish doesn't keep trying to process the
# request
- print STDERR "output directory $out_dir exists but is not a directory";
+ print STDERR "output directory $outdir exists but is not a directory";
stop_request_and_exit($req_id, $PS_EXIT_UNKNOWN_ERROR);
}
@@ -116,24 +119,24 @@
}
- # at this point we need to find out what kind of request type it is and
- # split the processing depending on the request type
- # the only processing required for a detectabilty query is to build the output fileset.
-
- # the following is for a postage stamp request
-
- my ($rlf, $reglist_name) = tempfile ("$out_dir/reglist.XXXX", UNLINK => !$save_temps);
+ # start building the list of files to be placed in the output fileset
+ my ($rlf, $reglist_name) = tempfile ("$outdir/reglist.XXXX", UNLINK => !$save_temps);
+
+ # results file
print $rlf "results.fits|||table|\n";
- print $rlf "results.mdc|||txt|\n";
+ # human readable representation of the results file
+ print $rlf "results.mdc|||text|\n";
my $err_file = "parse_error.txt";
- if (-e "$out_dir/$err_file" ) {
+ if (-e "$outdir/$err_file" ) {
print $rlf "$err_file|||text|\n";
}
- my ($tdf, $table_def_name) = tempfile ("$out_dir/tabledef.XXXX", UNLINK => !$save_temps);
-
+ # The results table definition file
+ my ($tdf, $table_def_name) = tempfile ("$outdir/tabledef.XXXX", UNLINK => !$save_temps);
# data for the header
print $tdf "$req_name|$req_id|\n";
+
+ # get the list of jobs generated for this request
my @jobs;
{
@@ -151,10 +154,13 @@
print STDERR "Request $req_id produced no jobs.\n"
}
- # This should not happen. A fake job should be queued
+ # This should not happen. A fake job should have been entered
stop_request_and_exit($req_id, $PS_EXIT_PROG_ERROR);
} else {
+if (0) {
my $metadata = $mdcParser->parse($output) or die("Unable to parse metdata config doc");
my $jobs = parse_md_list($metadata);
+}
+ my $jobs = parse_md_fast($mdcParser, $output);
@jobs = @$jobs;
@@ -163,5 +169,4 @@
my $exp_info;
- my $last_exp_id = -1;
foreach my $job (@jobs) {
my $job_id = $job->{job_id};
@@ -172,8 +177,7 @@
if (($fault eq $PSTAMP_DUP_REQUEST) and ($req_name eq "NULL")) {
- # this request had a duplicate request name. We can't put the results
- # on the data store since the product name is already used
- # maybe we could be more clever about this and choose a fileset name
- stop_request_and_exit($req_id, $fault);
+ # this request had a duplicate request name yet the parser didn't give
+ # it an "ERROR style name.
+ stop_request_and_exit($req_id, $PS_EXIT_PROG_ERROR);
}
my ($row, $req_info, $project) = get_request_info($rows, $rownum);
@@ -188,13 +192,22 @@
}
- if ($exp_id ne $last_exp_id) {
+ my $job_params = get_job_parameters($job);
+ my $stage = "";
+ if ($job_params) {
+ $stage = $job_params->{stage};
+ }
+
+ if ($stage ne 'stack') {
# get the metadata for the exposure (if any i.e. stack)
# returns an appropriate string if !$exp_id
- $exp_info = get_exposure_info($image_db, $exp_id);
- $last_exp_id = $exp_id;
+ $exp_info = get_exposure_info($job_params, $image_db, $exp_id);
+ } else {
+ my $filter = $job_params->{filter};
+ $filter = "0" if !$filter;
+ $exp_info = "0|0|0|$filter|0|0";
}
if (($job_type eq "stamp") || ($job_type eq "get_image") || ($job_type eq "none")) {
- my $jreglist = "$out_dir/reglist$job_id";
+ my $jreglist = "$outdir/reglist$job_id";
if (open JRL, "<$jreglist") {;
# process the reglist file to get the list of files produced by this job
@@ -213,7 +226,7 @@
# If not found check the PHU. If that doesn't work just set them to zero.
# XXX do this more cleanly
- my (undef, $ra_deg, $dec_deg) = split " ", `echo $out_dir/$img_name | fields -x 0 RA_DEG DEC_DEG`;
+ my (undef, $ra_deg, $dec_deg) = split " ", `echo $outdir/$img_name | fields -x 0 RA_DEG DEC_DEG`;
if (!defined $ra_deg) {
- (undef, $ra_deg, $dec_deg) = split " ", `echo $out_dir/$img_name | fields RA_DEG DEC_DEG`;
+ (undef, $ra_deg, $dec_deg) = split " ", `echo $outdir/$img_name | fields RA_DEG DEC_DEG`;
}
$ra_deg = 0.0 if (!$ra_deg);
@@ -246,5 +259,5 @@
# make the results file
{
- my $command = "$pstamp_results --input $table_def_name --output $out_dir/results.fits";
+ my $command = "$pstamp_results --input $table_def_name --output $outdir/results.fits";
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
run(command => $command, verbose => $verbose);
@@ -254,5 +267,5 @@
} else {
# dump a textual representation
- my $command = "$pstampdump $out_dir/results.fits > $out_dir/results.mdc";
+ my $command = "$pstampdump $outdir/results.fits > $outdir/results.mdc";
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
run(command => $command, verbose => $verbose);
@@ -266,6 +279,7 @@
# register the fileset
my $command = "$dsreg --list $reglist_name --add $fileset --product $product --type PSRESULTS";
- $command .= " --link --datapath $out_dir --ps0 $req_id";
- $command .= " --dbname $dbname" if $dbname;
+ $command .= " --link --datapath $outdir --ps0 $req_id";
+# XXX: let dsreg and config handle resolving dbname and dbserver
+# $command .= " --dbname $dbname" if $dbname;
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
@@ -280,5 +294,5 @@
# set the request's state to stop
{
- my $command = "$pstamptool -updatereq -req_id $req_id -state stop -fault $request_fault";
+ my $command = "$pstamptool -updatereq -req_id $req_id -set_state stop -set_fault $request_fault";
$command .= " -dbname $dbname" if $dbname;
$command .= " -dbserver $dbserver" if $dbserver;
@@ -296,5 +310,5 @@
my $fault = shift;
- my $command = "$pstamptool -updatereq -req_id $req_id -state stop -fault $fault";
+ my $command = "$pstamptool -updatereq -req_id $req_id -set_state stop -set_fault $fault";
$command .= " -dbname $dbname" if $dbname;
$command .= " -dbserver $dbserver" if $dbserver;
@@ -333,5 +347,5 @@
$data_group = "null" if !$data_group;
- # This is ugly, error prone and hard to change.
+ # XXX: This is ugly, error prone and hard to change.
# Create a results file module and provide a list of the names (we have the data in the columns)
my $rowinfo = "$row->{PROJECT}|$row->{JOB_TYPE}|$row->{REQ_TYPE}|$row->{IMG_TYPE}|";
@@ -344,55 +358,128 @@
}
+sub get_job_parameters {
+ my $job = shift;
+ if (!$job->{outputBase}) {
+ print "get_job_parameters: $job->{job_id} has no outputBase\n";
+ return undef;
+ }
+ my $params_file = $job->{outputBase} . '.mdc';
+ if (! -e $params_file ) {
+ print "get_job_parameters: $job->{job_id} has no parameters file\n";
+ return undef;
+ }
+ open IN, "<$params_file" or die "unable to open $params_file";
+ my $data = join "", ();
+ close IN;
+ if (! $data ) {
+ print "get_job_parameters: parameters file is empty\n";
+ return undef;
+ }
+ my $metadata = $mdcParser->parse($data) or die("Unable to parse metdata config doc");
+
+ # no need to use parse_md_fast here
+ my $results = parse_md_list($metadata);
+ if (scalar @$results != 1) {
+ print STDERR "get_job_params: failed to parse_md_list\n";
+ return undef;
+ }
+ return $results->[0];
+}
+
sub get_exposure_info {
+ my $job_params = shift;
my $image_db= shift;
my $exp_id = shift;
- if (!$exp_id or !$image_db) {
- # no exposure id just return zeros
- # XXX: we could put a value in for filter, but we don't have a good place to find it
-
- #"$mjd_obs|$ra_obs|$dec_obs|$filter|$exp_time|$fpa_id";
- return "0|0|0|0|0|0";
- }
-
- my $regtool = can_run('regtool') or (warn "Can't find regtool" and $missing_tools = 1);
- if ($missing_tools) {
- warn("Can't find required tools.");
- exit ($PS_EXIT_CONFIG_ERROR);
- }
-
- my $command = "$regtool -processedexp -dbname $image_db -exp_id $exp_id";
-
- # run the tool and parse the output
- my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
- run(command => $command, verbose => $verbose);
- unless ($success) {
- # not sure if we should die here
- die "cannot get exposure information for $exp_id from image database $image_db";
- }
- my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files
-
- my $output = join "", @$stdout_buf;
- if (!$output) {
- print STDERR "no output returned from $command\n" if $verbose;
- return undef;
- }
- my $metadata = $mdcParser->parse($output) or die("Unable to parse metdata config doc");
-
- my $exposures = parse_md_list($metadata);
- my $numExp = @$exposures;
-
- die "unexpected number of exposures $numExp found for exp_id: $exp_id in DB: $image_db" if $numExp != 1;
+ my ($dateobs, $ra, $decl, $filter, $exp_time, $exp_name);
+
+ $dateobs = $job_params->{dateobs};
+ $ra = $job_params->{ra};
+ $decl = $job_params->{decl};
+ $filter = $job_params->{filter};
+ $exp_time = $job_params->{exp_time};
+ $exp_name = $job_params->{exp_name};
+
+ unless (defined $dateobs and defined $ra and defined $decl and defined $filter and defined $exp_time and defined $exp_name) {
+ # job params don't have all of the values that we need (most likely this is a diff stage job)
+ # go look up the exposure if we have one
+ if (!$exp_id or !$image_db) {
+ # no exposure id just return zeros
+ # XXX: we could put a value in for filter, but we don't have a good place to find it
+
+ #"$mjd_obs|$ra_obs|$dec_obs|$filter|$exp_time|$fpa_id";
+ return "0|0|0|0|0|0";
+ }
+
+ my $exp;
+ # get cache of exposures for this image_db
+ my $exp_cache = $imagedb_cache{$image_db};
+ if (defined $exp_cache) {
+ $exp = $exp_cache->{$exp_id};
+ }
+
+ unless (defined $exp) {
+
+ my $regtool = can_run('regtool') or (warn "Can't find regtool" and $missing_tools = 1);
+ if ($missing_tools) {
+ warn("Can't find required tools.");
+ exit ($PS_EXIT_CONFIG_ERROR);
+ }
+
+ my $command = "$regtool -processedexp -dbname $image_db -exp_id $exp_id";
+
+ # run the tool and parse the output
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ # not sure if we should die here
+ die "cannot get exposure information for $exp_id from image database $image_db";
+ }
+ my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files
+
+ my $output = join "", @$stdout_buf;
+ if (!$output) {
+ print STDERR "no output returned from $command\n" if $verbose;
+ return undef;
+ }
+if (0) {
+ my $metadata = $mdcParser->parse($output) or die("Unable to parse metdata config doc");
+
+ my $exposures = parse_md_list($metadata);
+}
+ my $exposures = parse_md_fast($mdcParser, $output);
+ my $numExp = @$exposures;
+
+ die "unexpected number of exposures $numExp found for exp_id: $exp_id in DB: $image_db" if $numExp != 1;
+
+ $exp = $exposures->[0];
+
+ unless (defined $exp_cache) {
+ my %new_exp_cache;
+ $exp_cache = \%new_exp_cache;
+ $imagedb_cache{$image_db} = $exp_cache;
+ }
+ $exp_cache->{$exp_id} = $exp;
+ } else {
+ print "found $exp_id in cache\n";
+ }
+
+ #my $info = "$mjd_obs|$ra_obs|$dec_obs|$filter|$exp_time|$fpa_id";
+ $dateobs = $exp->{dateobs};
+ $ra = $exp->{ra};
+ $decl = $exp->{decl};
+ $filter = $exp->{filter};
+ $exp_time = $exp->{exp_time};
+ $exp_name = $exp->{exp_name};
+ }
+
+ die "failed to find exp_info for $exp_id" unless (defined $dateobs and defined $ra and defined $decl and defined $filter and defined $exp_time and defined $exp_name);
- my $exp = $exposures->[0];
-
- #my $info = "$mjd_obs|$ra_obs|$dec_obs|$filter|$exp_time|$fpa_id";
-
use constant RADIANS_TO_DEGREES => 90. / atan2(1, 0);
- my $ra_deg = $exp->{ra} * RADIANS_TO_DEGREES;
- my $decl_deg = $exp->{decl} * RADIANS_TO_DEGREES;
- my $mjd_obs = dateobs_to_mjd($exp->{dateobs});
-
- my $info = "$mjd_obs|$ra_deg|$decl_deg|$exp->{filter}|$exp->{exp_time}|$exp->{exp_name}";
+ my $ra_deg = $ra * RADIANS_TO_DEGREES;
+ my $decl_deg = $decl * RADIANS_TO_DEGREES;
+ my $mjd_obs = dateobs_to_mjd($dateobs);
+
+ my $info = "$mjd_obs|$ra_deg|$decl_deg|$filter|$exp_time|$exp_name";
return $info;
Index: branches/pap/pstamp/scripts/pstamp_get_image_job.pl
===================================================================
--- branches/pap/pstamp/scripts/pstamp_get_image_job.pl (revision 27708)
+++ branches/pap/pstamp/scripts/pstamp_get_image_job.pl (revision 28003)
@@ -87,5 +87,5 @@
}
-my $out_dir = dirname($output_base);
+my $outdir = dirname($output_base);
my $prefix = basename($output_base) . "_";
my $results_file = $output_base . ".bundle_results";
@@ -102,5 +102,5 @@
my $command = "$dist_bundle --camera $camera --stage $stage --stage_id $stage_id";
$command .= " --component $component";
- $command .= " --path_base $path_base --outdir $out_dir --results_file $results_file";
+ $command .= " --path_base $path_base --outdir $outdir --results_file $results_file";
$command .= " --prefix $prefix";
$command .= " --magicked" if $magicked;
@@ -139,5 +139,5 @@
}
-my $reglist = "$out_dir/reglist$job_id";
+my $reglist = "$outdir/reglist$job_id";
if (! open(REGLIST, ">$reglist") ) {
my_die("failed to open registration list: $reglist", $PS_EXIT_UNKNOWN_ERROR);
Index: branches/pap/pstamp/scripts/pstamp_job_run.pl
===================================================================
--- branches/pap/pstamp/scripts/pstamp_job_run.pl (revision 27708)
+++ branches/pap/pstamp/scripts/pstamp_job_run.pl (revision 28003)
@@ -74,4 +74,5 @@
my $ppstamp = can_run('ppstamp') or (warn "Can't find ppstamp" and $missing_tools = 1);
my $pstamp_get_image_job = can_run('pstamp_get_image_job.pl') or (warn "Can't find pstamp_get_image_job.pl" and $missing_tools = 1);
+my $dqueryparse = can_run('dqueryparse.pl') or (warn "Can't find dqueryparse.pl" and $missing_tools = 1);
if ($missing_tools) {
@@ -83,12 +84,19 @@
my $jobStatus;
if ($jobType eq "stamp") {
- my $argslist = "$outputBase.args";
- open ARGSLIST, "<$argslist" or my_die("failed to open argslist file $argslist", $job_id, $PS_EXIT_UNKNOWN_ERROR);
- my $argString = ;
- close ARGSLIST;
- chomp $argString;
+ my $params = read_params_file($outputBase);
+
+ my $argString;
+ $argString = $params->{job_args};
# XXX: should we do any other sanity checking?
- my_die("arglist file $argslist is empty", $job_id, $PS_EXIT_DATA_ERROR) if !$argString;
+ my_die("argument list is empty", $job_id, $PS_EXIT_DATA_ERROR) if !$argString;
+
+ # XXX: remove -astrom from argString and add it here
+
+ $argString .= " -file $params->{image}";
+ $argString .= " -mask $params->{mask}";
+ if ($options & $PSTAMP_SELECT_VARIANCE) {
+ $argString .= " -variance $params->{weight}";
+ }
my $command = "$ppstamp $outputBase $argString";
@@ -119,7 +127,7 @@
my %extensions = ( $PSTAMP_SELECT_IMAGE => "fits",
$PSTAMP_SELECT_MASK => "mk.fits",
- $PSTAMP_SELECT_WEIGHT => "wt.fits");
-
- my $output_mask = $options & ($PSTAMP_SELECT_IMAGE | $PSTAMP_SELECT_MASK | $PSTAMP_SELECT_WEIGHT);
+ $PSTAMP_SELECT_VARIANCE => "wt.fits");
+
+ my $output_mask = $options & ($PSTAMP_SELECT_IMAGE | $PSTAMP_SELECT_MASK | $PSTAMP_SELECT_VARIANCE);
foreach my $key (keys (%extensions)) {
@@ -139,5 +147,5 @@
}
- get_other_outputs($F, $outputBase, $options);
+ get_other_outputs($F, $outputBase, $options, $params);
close $F;
@@ -165,5 +173,30 @@
}
} elsif ($jobType eq "detect_query") {
- my_die("detect_query jobs not supported yet", $job_id,$PS_EXIT_CONFIG_ERROR);
+ # detect_query jobs are basically holders to note that we need a file updated before we can continue.
+ # Load the argument list that dqueryparse should have created the first time it ran, so we know how to
+ # run it again the same way.
+ my $argslist = "$outputBase/parse.args";
+ open ARGSLIST, "<$argslist" or my_die("failed to open argslist file $argslist", $job_id, $PS_EXIT_UNKNOWN_ERROR);
+ my $argString = ;
+ close ARGSLIST;
+ chomp $argString;
+
+ # XXX: should we do any other sanity checking?
+ my_die("arglist file $argslist is empty", $job_id, $PS_EXIT_DATA_ERROR) if !$argString;
+
+ my $command = "$dqueryparse --job_id $job_id $argString";
+ $command .= " --dbname $dbname" if $dbname;
+ $command .= " --dbserver $dbserver" if $dbserver;
+ $command .= " --verbose" if $verbose;
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+
+ # Although dqueryparse can potentially force more updates (and more jobs), it should still return success.
+ if ($success) {
+ $jobStatus = $PS_EXIT_SUCCESS;
+ } else {
+ $jobStatus = $error_code >> 8;
+ my_die("dqueryparse.pl failed with error code: $jobStatus", $job_id, $jobStatus);
+ }
} else {
my_die("unknown jobType $jobType found", $job_id, $PS_EXIT_PROG_ERROR);
@@ -172,6 +205,6 @@
# mark the job stopped in the database
{
- my $command = "$pstamptool -updatejob -job_id $job_id -state stop";
- $command .= " -fault $jobStatus" if $jobStatus;
+ my $command = "$pstamptool -updatejob -job_id $job_id -set_state stop";
+ $command .= " -set_fault $jobStatus" if $jobStatus;
$command .= " -dbname $dbname" if $dbname;
$command .= " -dbserver $dbserver" if $dbserver;
@@ -212,9 +245,12 @@
my $output_base = shift;
my $options = shift;
+ my $params = shift;
if ($options & ( $PSTAMP_SELECT_CMF | $PSTAMP_SELECT_PSF | $PSTAMP_SELECT_BACKMDL)) {
- my $comp = read_params_file($output_base);
-
- my $stage = $comp->{stage};
+ if (!$params) {
+ $params = read_params_file($output_base);
+ }
+
+ my $stage = $params->{stage};
# raw files don't have any other data products
@@ -225,8 +261,8 @@
# detected in pstampparse so that the user can be notified with
# a message in parse_error.txt ("warp do not have a background model")
- my $cmf_file = $comp->{cmf} if ($options & $PSTAMP_SELECT_CMF);
- my $psf_file = $comp->{psf} if ($options & $PSTAMP_SELECT_PSF);
- my $backmdl_file = $comp->{backmdl} if ($options & $PSTAMP_SELECT_BACKMDL);
- my $pattern_file = $comp->{pattern} if ($options & $PSTAMP_SELECT_BACKMDL);
+ my $cmf_file = $params->{cmf} if ($options & $PSTAMP_SELECT_CMF);
+ my $psf_file = $params->{psf} if ($options & $PSTAMP_SELECT_PSF);
+ my $backmdl_file = $params->{backmdl} if ($options & $PSTAMP_SELECT_BACKMDL);
+ my $pattern_file = $params->{pattern} if ($options & $PSTAMP_SELECT_BACKMDL);
my $outdir = dirname($output_base);
@@ -250,5 +286,5 @@
copy_and_register_file($f, $backmdl_file, $outdir, $prefix);
}
- if (0) {
+ if (0) {
# don't enable this yet
if ($pattern_file) {
@@ -256,5 +292,5 @@
copy_and_register_file($f, $pattern_file, $outdir, $prefix);
}
- }
+ }
}
}
@@ -317,7 +353,7 @@
my $command = "$pstamptool -updatejob";
$command .= " -job_id $job_id";
- $command .= " -fault $exit_code";
+ $command .= " -set_fault $exit_code";
# XXX: fix pstamptool to not require -state when -fault with nonzero value is provided
- $command .= " -state run";
+ $command .= " -set_state run";
$command .= " -dbname $dbname" if defined $dbname;
$command .= " -dbserver $dbserver" if defined $dbserver;
Index: branches/pap/pstamp/scripts/pstamp_parser_run.pl
===================================================================
--- branches/pap/pstamp/scripts/pstamp_parser_run.pl (revision 27708)
+++ branches/pap/pstamp/scripts/pstamp_parser_run.pl (revision 28003)
@@ -28,4 +28,6 @@
my $redirect_output;
my $product;
+my $outdir;
+my $label;
my $verbose;
my $dbname;
@@ -36,4 +38,6 @@
'uri=s' => \$uri,
'product=s' => \$product,
+ 'outdir=s' => \$outdir,
+ 'label=s' => \$label,
'redirect-output' => \$redirect_output,
'verbose' => \$verbose,
@@ -79,21 +83,25 @@
}
-# workdir is where all of the files generated for this request are placed
-# NOTE: this location needs to be kept in sync with the web interface ( request.php )
-my $datestr = strftime "%Y%m%d", gmtime;
-my $datedir = "$pstamp_workdir/$datestr";
-if (! -e $datedir ) {
- mkdir $datedir or my_die( "failed to create working directory $datedir for request id $req_id", $req_id,
+if (!$outdir or ($outdir eq "NULL")) {
+ # outdir is where all of the files generated for this request are placed
+ # NOTE: this location needs to be kept in sync with the web interface ( request.php )
+ my $datestr = strftime "%Y%m%d", gmtime;
+ my $datedir = "$pstamp_workdir/$datestr";
+ if (! -e $datedir ) {
+ mkdir $datedir or my_die( "failed to create working directory $datedir for request id $req_id", $req_id,
+ $PS_EXIT_CONFIG_ERROR);
+ }
+
+ $outdir = "$datedir/$req_id";
+}
+
+if (! -e $outdir ) {
+ mkdir $outdir or my_die("failed to create working directory $outdir for request id $req_id", $req_id,
$PS_EXIT_CONFIG_ERROR);
}
-
-my $workdir = "$datedir/$req_id";
-if (! -e $workdir ) {
- mkdir $workdir or my_die("failed to create working directory $workdir for request id $req_id", $req_id,
- $PS_EXIT_CONFIG_ERROR);
-}
+
if ($redirect_output) {
- my $logDest = "$workdir/psparse.$req_id.log";
+ my $logDest = "$outdir/psparse.$req_id.log";
$ipprc->redirect_output($logDest);
}
@@ -106,5 +114,5 @@
my $fn = basename($uri);
-my $new_uri = "$workdir/$fn";
+my $new_uri = "$outdir/$fn";
if ($uri =~ /^http:/) {
# if the uri is an http uri download the file
@@ -116,10 +124,11 @@
}
} elsif ($uri ne $new_uri) {
- # put a link to the file into the workdir
+ # copy the request file into outdir
if (-e $new_uri) {
+ # file exists already delete it incase the previous copy is bogus
unlink $new_uri or my_die("failed to unlink $new_uri", $req_id, $PS_EXIT_UNKNOWN_ERROR);
}
if (! copy $uri, $new_uri) {
- my_die ("failed to copy request file $uri to workdir $workdir", $req_id, $PS_EXIT_UNKNOWN_ERROR);
+ my_die ("failed to copy request file $uri to workdir $outdir", $req_id, $PS_EXIT_UNKNOWN_ERROR);
}
}
@@ -146,8 +155,10 @@
if ($request_type eq "PS1_PS_REQUEST") {
$reqType = 'pstamp';
- $parse_cmd = $pstampparse;
+ $parse_cmd = "$pstampparse";
+ $parse_cmd .= " --label $label" if $label;
} elsif ($request_type eq "MOPS_DETECTABILITY_QUERY") {
$reqType = 'dquery';
- $parse_cmd = $dqueryparse;
+ $parse_cmd = "$dqueryparse";
+ $parse_cmd .= " --label $label" if $label;
} else {
print STDERR "Unknown request type $request_type found in $uri";
@@ -169,7 +180,7 @@
# that they sent us a request file that we don't understand
- my $command = "$pstamptool -updatereq -req_id $req_id -state run";
- $command .= " -reqType unknown";
- $command .= " -fault $PS_EXIT_DATA_ERROR";
+ my $command = "$pstamptool -updatereq -req_id $req_id -set_state run";
+ $command .= " -set_reqType unknown";
+ $command .= " -set_fault $PS_EXIT_DATA_ERROR";
$command .= " -dbname $dbname" if $dbname;
$command .= " -dbserver $dbserver" if $dbserver;
@@ -182,5 +193,5 @@
}
-$parse_cmd .= " --mode queue_job --req_id $req_id --product $product --out_dir $workdir --file $uri";
+$parse_cmd .= " --mode queue_job --req_id $req_id --product $product --outdir $outdir --file $uri";
$parse_cmd .= " --dbname $dbname" if $dbname;
$parse_cmd .= " --dbserver $dbserver" if $dbserver;
@@ -190,5 +201,5 @@
my $fault;
{
- my $error_file_name = "$workdir/parse_error.txt";
+ my $error_file_name = "$outdir/parse_error.txt";
# get rid of any error file from previous attempt to parse this request
unlink $error_file_name if (-e $error_file_name);
@@ -215,5 +226,5 @@
unless ($success) {
$fault = $error_code >> 8;
- $newState = "stop";
+# $newState = "stop";
}
}
@@ -223,8 +234,9 @@
#
{
- my $command = "$pstamptool -updatereq -req_id $req_id -state $newState";
- $command .= " -reqType $reqType" if $reqType;
- $command .= " -uri $new_uri" if $new_uri;
- $command .= " -fault $fault" if $fault;
+ my $command = "$pstamptool -updatereq -req_id $req_id -set_state $newState";
+ $command .= " -set_outdir $outdir";
+ $command .= " -set_reqType $reqType" if $reqType;
+ $command .= " -set_uri $new_uri" if $new_uri;
+ $command .= " -set_fault $fault" if $fault;
$command .= " -dbname $dbname" if $dbname;
$command .= " -dbserver $dbserver" if $dbserver;
@@ -265,5 +277,5 @@
carp($msg);
- my $command = "$pstamptool -updatereq -req_id $req_id -fault $fault";
+ my $command = "$pstamptool -updatereq -req_id $req_id -set_fault $fault";
$command .= " -dbname $dbname" if $dbname;
$command .= " -dbserver $dbserver" if $dbserver;
Index: branches/pap/pstamp/scripts/pstamp_request_file
===================================================================
--- branches/pap/pstamp/scripts/pstamp_request_file (revision 27708)
+++ branches/pap/pstamp/scripts/pstamp_request_file (revision 28003)
@@ -187,5 +187,11 @@
for (my $i = 0; $i < scalar @colNames; $i++) {
my $writeType = $colWriteType[$i];
- $outFits->write_col( $writeType, $i + 1, 1, 1, $numRows, $colData->[$i], $status );
+ my $data = $colData->[$i];
+ if ($writeType == TULONG) {
+ die "invalid integer data found in column $i\n" unless validIntegers($data);
+ } elsif ($writeType == TDOUBLE) {
+ die "invalid numeric data found in column $i\n" unless validNumbers($data);
+ }
+ $outFits->write_col( $writeType, $i + 1, 1, 1, $numRows, $data, $status );
check_fitsio( $status );
}
@@ -312,4 +318,42 @@
}
+sub checkValid {
+ my $aref = shift;
+ my $float = shift;
+
+ return 0 if !defined $aref;
+
+ my $valid = 0;
+ my $row = 0;
+ foreach my $val (@$aref) {
+ $row++;
+ if ($float) {
+ if (!($val =~ /^([+-]?)(?=\d|\.\d)\d*(\.\d*)?([Ee]([+-]?\d+))?$/)) {
+ $valid = 0;
+ print STDERR "Error on row $row: '$val' is not a valid number\n";
+ last;
+ }
+ } else {
+ if ($val =~ /\D/) {
+ $valid = 0;
+ print STDERR "Error on row $row: '$val' is not a valid integer\n";
+ last;
+ }
+ }
+ $valid = 1;
+ }
+
+ return $valid;
+}
+sub validIntegers {
+ return checkValid(@_, 0);
+}
+
+sub validNumbers
+{
+ return checkValid(@_, 1);
+}
+
+
sub printhelp
{
Index: branches/pap/pstamp/scripts/pstampparse.pl
===================================================================
--- branches/pap/pstamp/scripts/pstampparse.pl (revision 27708)
+++ branches/pap/pstamp/scripts/pstampparse.pl (revision 28003)
@@ -17,4 +17,5 @@
use Carp;
use POSIX;
+use Time::HiRes qw(gettimeofday);
my $verbose;
@@ -24,6 +25,7 @@
my $request_file_name;
my $mode = "list_uri";
-my $out_dir;
+my $outdir;
my $product;
+my $label;
my $save_temps;
my $no_update;
@@ -32,6 +34,7 @@
'file=s' => \$request_file_name,
'req_id=s' => \$req_id,
- 'out_dir=s' => \$out_dir,
+ 'outdir=s' => \$outdir,
'product=s' => \$product,
+ 'label=s' => \$label,
'mode=s' => \$mode,
'dbname=s' => \$dbname,
@@ -42,10 +45,10 @@
);
-die "invalid mode '$mode'" unless ($mode eq "list_uri") or ($mode eq "list_job") or ($mode eq "queue_job");
+die "invalid mode '$mode'" unless ($mode eq "list_uri") or ($mode eq "queue_job");
die "--file is required" if !defined($request_file_name);
if ($mode ne "list_uri") {
die "req_id is required" if !$req_id;
- die "out_dir is required" if !$out_dir;
+ die "outdir is required" if !$outdir;
die "product is required" if !$product;
}
@@ -64,5 +67,4 @@
my $pstamptool = can_run('pstamptool') or (warn "Can't find pstamptool" and $missing_tools = 1);
my $pstampdump = can_run('pstampdump') or (warn "Can't find pstampdump" and $missing_tools = 1);
-my $dvoImagesAtCoords = can_run('dvoImagesAtCoords') or (warn "Can't find dvoImagesAtCoords" and $missing_tools = 1);
my $fields = can_run('fields') or (warn "Can't find fields" and $missing_tools = 1);
@@ -127,6 +129,6 @@
# update the database with the request name. This will be used as the
# the output data store's product name
- my $command = "$pstamptool -updatereq -req_id $req_id -name $req_name";
- $command .= " -outProduct $product";
+ my $command = "$pstamptool -updatereq -req_id $req_id -set_name $req_name";
+ $command .= " -set_outProduct $product";
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
run(command => $command, verbose => $verbose);
@@ -145,4 +147,5 @@
my $rows;
{
+ my $start_request_file = gettimeofday();
my $command = "$pstampdump $request_file_name";
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
@@ -152,11 +155,17 @@
}
if (@$stdout_buf) {
- my $table = $mdcParser->parse(join "", @$stdout_buf) or
- my_die("Unable to parse metdata config doc", $PS_EXIT_UNKNOWN_ERROR);
- $rows = parse_md_list($table);
- }
-
-}
-
+ $rows = parse_md_fast($mdcParser, join "", @$stdout_buf);
+ }
+ my $dtime_request_file = gettimeofday() - $start_request_file;
+ print "Time to read and parse request file: $dtime_request_file\n";
+
+}
+
+#
+# Loop over rows in the request file collecting consecutive rows that have the "same images of interest"
+# in the sense that their selection parameters will yield the same "Runs".
+# Process the groups of rows together to reduce lookup time and to potentially make multiple
+# stamps from the same ppstamp process.
+#
my @rowList;
my $num_jobs = 0;
@@ -165,30 +174,90 @@
my $need_magic;
foreach my $row (@$rows) {
- # XXX: TODO: sanity check all parameters
-
- # If we encounter an error for a particular row add a job with the proper fault code.
- # If we encounter an error in this loop we shouldn't really just die.
- # We only do that now in the case of I/O or DB errors or the like.
-
- my $rownum = $row->{ROWNUM};
- my $job_type = $row->{JOB_TYPE};
-
- # parameters that select the images of interest
- my $project = $row->{PROJECT};
-
- # note: resolve_project avoids running pstamptool every time by remembering the
- # last project resolved
- my $proj_hash = resolve_project($ipprc, $project, $dbname, $dbserver);
- if (!$proj_hash) {
- print STDERR "project $project not found\n" if $verbose;
- insertFakeJobForRow($row, 1, $PSTAMP_UNKNOWN_PRODUCT);
+ # santiy check the paramaters
+ if (!checkRow($row)) {
+ # when it enconters an error checkRow adds a fake job with an appropriate error code to the database
$num_jobs++;
next;
}
+ # initialize counter for "job number"
+ $row->{job_num} = 0;
+ $row->{error_code} = 0;
+
+ if (scalar @rowList == 0) {
+ push @rowList, $row;
+ next;
+ }
+
+ my $firstRow = $rowList[0];
+ if (same_images_of_interest($firstRow, $row)) {
+ # add this row to the list and move on
+ push @rowList, $row;
+ next;
+ }
+
+ # the images of interest for this new row doesn't match the list.
+ # process the list ...
+ $num_jobs += processRows(\@rowList);
+
+ # and reset the list to contain just the new row
+ @rowList = ($row);
+}
+
+# out of rows process the list
+if (scalar @rowList > 0) {
+ $num_jobs += processRows(\@rowList);
+}
+
+if (($mode eq "queue_job") and ($num_jobs eq 0)) {
+ print STDERR "no jobs created for $req_name\n" if $verbose;
+ insertFakeJobForRow(undef, 0, $PSTAMP_INVALID_REQUEST);
+}
+
+exit 0;
+
+sub checkRow {
+
+ my $row = shift;
+
+ # If we encounter an error for a particular row add a job with the proper fault code.
+
+ my $rownum = $row->{ROWNUM};
+ if (!validID($rownum)) {
+ print STDERR "$rownum is not a valid ROWNUM\n";
+ insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
+ return 0;
+ }
+ my $job_type = $row->{JOB_TYPE};
+ if (($job_type ne "stamp") and ($job_type ne "get_image")) {
+ print STDERR "$job_type is not a valid JOB_TYPE\n";
+ insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
+ return 0;
+ }
+
my $req_type = $row->{REQ_TYPE};
- $stage = $row->{IMG_TYPE};
- my $id = $row->{ID};
+ if (($req_type ne "byid") and ($req_type ne "bycoord") and ($req_type ne "byexp") and
+ ($req_type ne "byskycell") and ($req_type ne "bydiff")) {
+ print STDERR "$req_type is not a valid REQ_TYPE\n";
+ insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
+ return 0;
+ }
+ if ($job_type eq 'get_image') {
+ unless ($req_type eq 'byid' or $req_type eq 'byexp') {
+ print STDERR "REQ_TYPE must be 'byid' or 'byexp' for JOB_TYPE 'get_image'\n";
+ insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
+ return 0;
+ }
+ }
+
+
my $component = $row->{COMPONENT};
- my $tess_id = $row->{TESS_ID};
+ if (!defined $component or (lc($component) eq "null") or (lc($component) eq "all")) {
+ if ($job_type eq 'get_image') {
+ $row->{COMPONENT} = 'all';
+ } else {
+ $row->{COMPONENT} = $component = "";
+ }
+ }
+ $row->{TESS_ID} = "" if !defined $row->{TESS_ID};
my $filter = $row->{REQFILT};
@@ -196,9 +265,19 @@
if (length($filter) == 1) {
# allow single character filter cuts to work
- $filter .= '%';
+ $row->{REQFILT} .= '%';
}
}
my $mjd_min = $row->{MJD_MIN};
+ if (defined($mjd_min) and !validNumber($mjd_min)) {
+ print STDERR "$mjd_min is not a valid MJD_MIN\n";
+ insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
+ return 0;
+ }
my $mjd_max = $row->{MJD_MAX};
+ if (defined($mjd_max) and !validNumber($mjd_max)) {
+ print STDERR "$mjd_max is not a valid MJD_MAX\n";
+ insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
+ return 0;
+ }
my $data_group = $row->{DATA_GROUP};
if (!defined $data_group) {
@@ -209,9 +288,7 @@
}
- my $x = $row->{CENTER_X};
- my $y = $row->{CENTER_Y};
-
- # XXX things don't work if bit zero of option mask is not set;
+ # req_finish doesn't work if bit zero of option mask is not set;
$row->{OPTION_MASK} |= 1;
+
my $option_mask= $row->{OPTION_MASK};
my $inverse = ($option_mask & $PSTAMP_SELECT_INVERSE) ? 1 : 0;
@@ -222,124 +299,148 @@
my $skycenter = $row->{skycenter} = ! ($row->{COORD_MASK} & $PSTAMP_CENTER_IN_PIXELS);
- my $search_component = (!defined($component) or ($component eq "null")) ? "" : $component;
-
- if (!$skycenter and !$search_component) {
- print STDERR "COMPONENT must be specified for pixel coordinate ROI center\n" if $verbose;
+ if (!$skycenter and !$component) {
+ print STDERR "COMPONENT must be specified for pixel coordinate ROI center\n";
insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
- $num_jobs++;
- next;
- }
-
- # user requested us to search all components. Set to ""
- $search_component = "" if $search_component eq "all";
-
+ return 0;
+ }
+
+ my $stage = $row->{IMG_TYPE};
if (!check_image_type($stage)) {
- print STDERR "invalid IMG_TYPE for row $rownum\n" if $verbose;
+ print STDERR "invalid IMG_TYPE for row $rownum\n";
insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
- $num_jobs++;
- next;
+ return 0;
}
if ((($job_type eq "stamp") or ($req_type eq "bycoord")) and ! validROI($row)) {
- print STDERR "invalid ROI for row $rownum\n" if $verbose;
+ print STDERR "invalid ROI for row $rownum\n";
insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
- $num_jobs++;
- next;
+ return 0;
}
if (($req_type eq "byexp") and ($stage eq "stack")) {
- print STDERR "byexp not implemented for stack stage. row: $rownum\n" if $verbose;
+ print STDERR "byexp not implemented for stack stage. row: $rownum\n";
insertFakeJobForRow($row, 1, $PSTAMP_NOT_IMPLEMENTED);
- $num_jobs++;
- next;
- }
-
-
+ return 0;
+ }
+
# $mode list_uri is a debugging mode (it may used by the http interface)
# if this happens just croak
- my_die("job_type is list_uri but mode is $mode", $PS_EXIT_PROG_ERROR) if ($job_type eq "list_uri") and ($mode ne "list_uri");
-
+ # my_die("job_type is list_uri but mode is $mode", $PS_EXIT_PROG_ERROR) if ($job_type eq "list_uri") and ($mode ne "list_uri");
+
+
+ if ($req_type eq "bycoord") {
+ if (!$skycenter) {
+ print STDERR "center must be specified in sky coordintes for bycoord";
+ insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
+ return 0;
+ }
+ }
+
+ if (($req_type eq "byid") or ($req_type eq "bydiff")) {
+ if (!validID($row->{ID})) {
+ print STDERR "ID must be a positive integer for req_type $req_type\n";
+ insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
+ return 0
+ }
+ }
+
+ return 1;
+}
+
+sub processRows {
+ my $rowList = shift;
+ my $num_jobs = 0;
+
+ # all rows in the list are compatible
+ my $row = $rowList->[0];
+
+ my $project = $row->{PROJECT};
+
+ # note: resolve_project avoids running pstamptool every time by remembering the
+ # last project resolved
+ my $proj_hash = resolve_project($ipprc, $project, $dbname, $dbserver);
+ if (!$proj_hash) {
+ print STDERR "project $project not found\n" ;
+ foreach $row (@$rowList) {
+ insertFakeJobForRow($row, 1, $PSTAMP_UNKNOWN_PRODUCT);
+ $num_jobs++;
+ }
+ return $num_jobs;
+ }
+ my $req_type = $row->{REQ_TYPE};
+ $stage = $row->{IMG_TYPE};
+ my $id = $row->{ID};
+ my $component = $row->{COMPONENT};
+ my $tess_id = $row->{TESS_ID};
+
+ my $filter = $row->{REQFILT};
+ my $mjd_min = $row->{MJD_MIN};
+ my $mjd_max = $row->{MJD_MAX};
+ my $data_group = $row->{DATA_GROUP};
+
+ my $rownum = $row->{ROWNUM};
+ my $job_type = $row->{JOB_TYPE};
+ my $option_mask= $row->{OPTION_MASK};
+
my $image_db = $proj_hash->{dbname};
my $camera = $proj_hash->{camera};
- $need_magic = $proj_hash->{need_magic};
+ $need_magic = $proj_hash->{need_magic};
# Temporary hack so that MOPS can get at non-magicked data
-# if ($product and ($product eq "mops-pstamp-results")) {
-# $need_magic = 0;
-# }
-
- # For "stamp" and "list_uri" jobs collect rows with the same images of interest in a list so that they
- # can be looked up together.
- if (@rowList) {
- my $firstRow = $rowList[0];
- if (($firstRow->{JOB_TYPE} ne "get_image") and same_images_of_interest($firstRow, $row)) {
-
- # add this row to the list and move on
- push @rowList, $row;
-
- next;
-
- } else {
- # this row has different selectors
- # queue the jobs for the ones we've collected
- $num_jobs += queueJobs($mode, \@rowList, $imageList);
- @rowList = ();
- }
- }
-
- # look up images for the current row
- if ($req_type eq "bycoord") {
- if (!$skycenter) {
- print STDERR "center must be specified in sky coordintes for bycoord" if $verbose;
- insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
- $num_jobs++;
- next;
- }
- }
- if (($req_type eq "byid") or ($req_type eq "bydiff")) {
- if (!validID($id)) {
- print STDERR "ID must be a positive integer for req_type $req_type\n" if $verbose;
- insertFakeJobForRow($row, 1, $PSTAMP_INVALID_REQUEST);
- $num_jobs++;
- next;
- }
- }
+ my $allow_mops_unmagicked = 1;
+ if ($allow_mops_unmagicked) {
+ if ($product and (($product eq "mops-pstamp-results") or
+ ($product eq "mops-pstamp-results2"))) {
+ $need_magic = 0;
+ }
+ }
+
+ $need_magic = 0 if $stage eq 'stack';
+
+ my $numRows = scalar @$rowList;
+
+ print "Collected $numRows rows beginning with row $rownum. $req_type $stage $id $tess_id $component\n";
+
# Call PS::IPP::PStamp::Job locate_images subroutine to get the images for this
# request specification. An array reference is returned.
- $imageList = locate_images($ipprc, $image_db, $req_type, $stage, $id, $tess_id, $search_component,
- $option_mask, $need_magic, $x, $y, $mjd_min, $mjd_max, $filter, $data_group, $verbose);
-
- if (!$imageList or !@$imageList) {
- print STDERR "no matching images found for row $rownum\n" if $verbose;
- # note in this case queueJobs inserts the fake job for these rows
- }
+ my $start_locate = gettimeofday();
+
+ # XXX: perhaps we should get rid of most of this argument list.
+ # Now that we are passing down compatible rows all of the
+ # information required is contained there
+
+ $imageList = locate_images($ipprc, $image_db, \@rowList, $req_type, $stage, $id, $tess_id, $component,
+ $option_mask, $need_magic, $mjd_min, $mjd_max, $filter, $data_group, $verbose);
+
+ my $dtime_locate = gettimeofday() - $start_locate;
+ print "Time to locate_images for row $rownum $dtime_locate\n";
+
+ # handle this XXX: what did I mean by this comment
$row->{need_magic} = $need_magic;
- push @rowList, $row;
-}
-
-if (@rowList) {
+
$num_jobs += queueJobs($mode, \@rowList, $imageList);
-}
-
-if (($mode eq "queue_job") and ($num_jobs eq 0)) {
- print STDERR "no jobs created for $req_name\n" if $verbose;
- insertFakeJobForRow(undef, 0, $PSTAMP_INVALID_REQUEST);
-}
-
-# PAU
-
-exit 0;
-
-
-sub queueJobsForRow
+
+ # if a row slipped through with no jobs add one
+ foreach my $row (@rowList) {
+ if ($row->{job_num} == 0) {
+ print "row $row->{ROWNUM} produced no jobs\n";
+ print STDERR "row $row->{ROWNUM} produced no jobs\n";
+ my $error_code = $row->{error_code};
+ $error_code = $PSTAMP_NO_IMAGE_MATCH if !$error_code;
+ insertFakeJobForRow($row, ++$row->{job_num}, $error_code);
+ }
+ }
+
+ return $num_jobs;
+}
+
+sub queueJobForImage
{
my $row = shift;
my $stage = shift;
- my $imageList = shift;
- my $have_skycells = shift;
+ my $image = shift;
my $need_magic = shift;
my $mode = shift;
- my $num_jobs = 0;
my $rownum = $row->{ROWNUM};
my $option_mask = $row->{OPTION_MASK};
@@ -365,28 +466,227 @@
}
+ my $component = $image->{component};
+
+ my $job_num = ++($row->{job_num});
+
+ my $imagefile = $image->{image};
+ if (($stage ne "stack") and ($need_magic and !$image->{magicked})) {
+ # XXX: should we add a faulted job so the client can know what happened if no images come back?
+ # The test for destreaked is made in locate_images now so this code never runs. This leads to no feedback
+ # to users, but speeds up processing significantly
+ print STDERR "skipping non-magicked image $imagefile\n" if $verbose;
+
+ # for now assume yes.
+
+ insertFakeJobForRow($row, $job_num, $PSTAMP_NOT_DESTREAKED);
+ return 1;
+ } elsif ($stage eq "stack") {
+ # unconvolved stack images weren't available prior to some point in time.
+ # XXX: handle this more correctly by examining the stack run's config dump file.
+ # It looks like # the feature was turned on sometime around November 11, 2009. stackRun 30067 is the lowest
+ # one that I found with an unconvolved image.
+ my $MIN_GPC1_STACK_ID_WITH_UNCONVOLVED_IMAGES = 30067;
+ if ($row->{unconvolved} and ($row->{PROJECT} eq 'gpc1') and
+ ($image->{stack_id} < $MIN_GPC1_STACK_ID_WITH_UNCONVOLVED_IMAGES)) {
+ print STDERR "Unconvolved stack image is not available for stackRun.stack_id: $image->{stack_id}\n";
+ insertFakeJobForRow($row, $job_num, $PSTAMP_NOT_AVAILABLE);
+ return 1;
+ }
+ }
+ my $exp_id = $image->{exp_id};
+
+ my $args = $roi_string ? $roi_string : "";
+ if ($stage eq "raw" or $stage eq "chip") {
+ $args .= " -class_id $component" if $component;
+ }
+
+ # add astrometry file for raw and chip images if one is available
+ if (($stage eq "chip") || ($stage eq "raw")) {
+ $args .= " -astrom $image->{astrom}" if $image->{astrom};
+ }
+
+ $image->{job_args} = $args;
+
+ my $base = basename($image->{image});
+ if (! $base =~ /.fits$/ ) {
+ my_die("unexpected image file name found $image->{image}", $PS_EXIT_PROG_ERROR);
+ }
+ $base =~ s/.fits$//;
+
+ my $output_base = "$outdir/${rownum}_${job_num}_${base}";
+ write_params($output_base, $image);
+
+ my $newState = "run";
+ my $fault = 0;
+ my $dep_id;
+
+ # XXX: this code is repeated in queueGetImageJobs we should encapsulate it in a subroutine and share it
+ if ($stage ne 'raw') {
+ # updates for stack stage not supported yet
+ my $allow_wait_for_update = ($stage ne 'stack');
+ my $run_state = $image->{state};
+ my $data_state = $image->{data_state};
+ $data_state = $run_state if $stage eq 'stack';
+ if (($run_state eq 'goto_purged') or ($data_state eq 'purged') or
+ ($run_state eq 'drop') or
+ ($run_state eq 'error_cleaned') or ($data_state eq 'error_cleaned') or
+ ($run_state eq 'goto_scrubbed') or ($data_state eq 'scrubbed')) {
+ # image is gone and it's not coming back
+ $newState = 'stop';
+ $fault = $PSTAMP_GONE;
+ } elsif (($data_state ne 'full') or ($need_magic and ($image->{magicked} < 0))) {
+ if ($stage eq 'chip') {
+ my $burntool_state = $image->{burntool_state};
+ if ($burntool_state and (abs($burntool_state) < 14)) {
+ $newState = 'stop';
+ $fault = $PSTAMP_NOT_AVAILABLE;
+ }
+ }
+ if (!$allow_wait_for_update) {
+ print STDERR "wait for update not supported for stage $stage yet\n";
+ $newState = 'stop';
+ $fault = $PSTAMP_NOT_AVAILABLE;
+ }
+ if (!$fault) {
+ # wait for update unless the customer asks us not to
+ if (($option_mask & $PSTAMP_NO_WAIT_FOR_UPDATE)) {
+ $newState = 'stop';
+ $fault = $PSTAMP_NOT_AVAILABLE;
+ } elsif (!$image->{magicked}) {
+ $newState = 'stop';
+ $fault = $PSTAMP_NOT_DESTREAKED;
+ } else {
+ # cause the image to be re-made
+ # set up to queue an update run
+ queue_update_run(\$newState, \$fault, \$dep_id, $image->{imagedb},
+ $run_state, $stage, $image->{stage_id}, $image->{component}, $need_magic);
+ }
+ }
+ }
+ }
+
+ my $command = "$pstamptool -addjob -req_id $req_id -job_type $row->{JOB_TYPE}"
+ . " -outputBase $output_base -rownum $rownum -state $newState -options $option_mask";
+ $command .= " -fault $fault" if $fault;
+ $command .= " -exp_id $exp_id" if $exp_id;
+ $command .= " -dep_id $dep_id" if $dep_id;
+
+ if (!$no_update) {
+ # mode eq "queue_job"
+ my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
+ run(command => $command, verbose => $verbose);
+ unless ($success) {
+ print STDERR @$stderr_buf;
+ # XXX TODO: now what? Should we mark the error state for the request?
+ # should we keep going for other uris? If so how do we report that some
+ # of the work that the request wanted isn't going to get done
+ my_die("failed to queue job for request $req_id", $PS_EXIT_UNKNOWN_ERROR);
+ }
+ } else {
+ print "skipping command: $command\n";
+ }
+
+ return 1;
+}
+
+# queue jobs for a collection of request specifications that have the same Images of Interest
+sub queueJobs
+{
+ my $mode = shift;
+ my $rowList = shift;
+ my $imageList = shift;
+
+ my $firstRow = $rowList[0];
+ my $stage = $firstRow->{IMG_TYPE};
+ my $job_type = $firstRow->{JOB_TYPE};
+ my $need_magic = $firstRow->{need_magic};
+
+ my $num_jobs = 0;
+
+ if ($mode eq "list_uri") {
+ foreach my $image (@$imageList) {
+ print "$image->{image}\n";
+ }
+ } elsif ($job_type eq "get_image") {
+ my $n = scalar @$rowList;
+
+ my_die( "error: unexpected number of rows for get_image request: $n", $PS_EXIT_PROG_ERROR) if $n != 1;
+
+ $num_jobs = queueGetImageJobs($firstRow, $imageList, $stage, $need_magic, $mode);
+
+ } else {
+ if (!$imageList or (scalar @$imageList eq 0)) {
+ # We didn't find any images for this set of rows. Insert a fake job to carry
+ # the status back to the requestor.
+ foreach my $row (@$rowList) {
+ my $error_code = $row->{error_code};
+ $error_code = $PSTAMP_NO_IMAGE_MATCH if !$error_code;
+ insertFakeJobForRow($row, ++$row->{job_num}, $error_code);
+ $num_jobs++;
+ }
+ return $num_jobs;
+ }
+
+ foreach my $image (@$imageList) {
+ # get the array of row indices that touch this image
+ my $row_index = $image->{row_index};
+ if (!$row_index or scalar @$row_index == 0) {
+ # XXX should this happen? Why did something get returned.
+ print "image ${stage}_id: $image->{stage_id} component: $image->{component} matched no rows\n";
+ next;
+ }
+ # XXX: TODO: eventually we may change ppstamp to be able to make multiple stamps per invocation
+
+ foreach my $i (@$row_index) {
+ my $row = $rowList->[$i];
+
+ $num_jobs += queueJobForImage($row, $stage, $image, $need_magic, $mode);
+ }
+ }
+ }
+
+ return $num_jobs;
+}
+
+sub queueGetImageJobs
+{
+ my $row = shift;
+ my $imageList = shift;
+ my $stage = shift;
+ my $need_magic = shift;
+ my $mode = shift;
+
+ my $num_jobs = 0;
+ my $rownum = $row->{ROWNUM};
+ my $option_mask = $row->{OPTION_MASK};
+
+ # For dist_bundle we need
+ # --camera from $image
+ # --stage
+ # --stage_id from $image
+ # --component from $image
+ # --path_base
+ # --outdir global to this script
+
# loop over images
- my $job_num = 0;
foreach my $image (@$imageList) {
- my $component;
- if ($have_skycells) {
- $component = $image->{skycell_id};
- } else {
- $component = $image->{class_id};
- }
-
- # skip this component if it is not in the list for this row
- next if ! $components->{$component};
-
- $job_num++;
+ my $stage_id = $image->{stage_id};
+ my $component = $image->{component};
+
+ # skip faulted components for now. Should we even be here?
+ if ($image->{fault} > 0) {
+ printf STDERR "skipping faulted component for $stage $stage_id $component\n" if $verbose;
+ next;
+ }
+
+ my $job_num = ++($row->{job_num});
my $imagefile = $image->{image};
if (($stage ne "stack") and ($need_magic and !$image->{magicked})) {
- # XXX: should we add a faulted job so the client can know what happened if no images come back?
- # The test for destreaked is made in locate_images now so this code never runs. This leads to no feedback
- # to users, but speeds up processing significantly
+ # we only get here if req_type is (byid or byexp). For other types the test for magicked is performed
+ # in locate_images because it's much more efficient to do the test in the database.
+ # For these two modes we fall through to here in order to give feedback to the requestor as
+ # to why the request failed to queue jobs.
print STDERR "skipping non-magicked image $imagefile\n" if $verbose;
-
- # for now assume yes.
-
insertFakeJobForRow($row, $job_num, $PSTAMP_NOT_DESTREAKED);
$num_jobs++;
@@ -396,46 +696,5 @@
my $exp_id = $image->{exp_id};
- my $args = $roi_string ? $roi_string : "";
- if ($stage eq "raw" or $stage eq "chip") {
- $args .= " -class_id $component" if $component;
- }
-
- # add astrometry file for raw and chip images if one is available
- if (($stage eq "chip") || ($stage eq "raw")) {
- $args .= " -astrom $image->{astrom}" if $image->{astrom};
- }
-
- $image->{job_args} = $args;
-
- # XXX: we can get rid of the following everything that we need is
- # in the params file
-
- $args .= " -file $imagefile";
-
- if (($option_mask & $PSTAMP_SELECT_MASK) && $image->{mask} ) {
- $args .= " -mask $image->{mask}";
- }
- if (($option_mask & $PSTAMP_SELECT_WEIGHT) and $image->{weight} ) {
- $args .= " -variance $image->{weight}";
- }
-
- my $base = basename($image->{image});
- if (! $base =~ /.fits$/ ) {
- my_die("unexpected image file name found $image->{image}", $PS_EXIT_PROG_ERROR);
- }
- $base =~ s/.fits$//;
-
- # XXX: TODO use filerule for this. I don't have a camera defined here
- if (($stage eq 'chip') and ($image->{camera} eq 'GPC1')) {
- $base = "${base}.${component}";
- }
-
- my $output_base = "$out_dir/${rownum}_${job_num}_${base}";
- my $argslist = "${output_base}.args";
-
- # copy the argument list to a file
- open ARGSLIST, ">$argslist" or my_die("failed to open $argslist", $PS_EXIT_UNKNOWN_ERROR);
- print ARGSLIST "$args\n";
- close ARGSLIST or my_die("failed to close $argslist", $PS_EXIT_UNKNOWN_ERROR);
+ my $output_base = "$outdir/${rownum}_${job_num}";
write_params($output_base, $image);
@@ -454,7 +713,7 @@
$newState = 'stop';
$fault = $PSTAMP_GONE;
- } elsif (($data_state ne 'full') or ($run_state ne 'full' )) {
- # don't wait for update unless the caller asks us to
- if (!($option_mask & $PSTAMP_WAIT_FOR_UPDATE)) {
+ } elsif (($data_state ne 'full') or ($need_magic and ($image->{magicked} < 0))) {
+ # wait for update unless the customer asks us to not to
+ if ($option_mask & $PSTAMP_NO_WAIT_FOR_UPDATE) {
$newState = 'stop';
$fault = $PSTAMP_NOT_AVAILABLE;
@@ -462,6 +721,6 @@
# cause the image to be re-made
# set up to queue an update run
- queue_update_run(\$newState, \$fault, \$dep_id, $image->{image_db},
- $run_state, $stage, $image->{stage_id}, $need_magic);
+ queue_update_run(\$newState, \$fault, \$dep_id, $image->{imagedb},
+ $run_state, $stage, $image->{stage_id}, $image->{component}, $need_magic);
}
}
@@ -475,256 +734,5 @@
$command .= " -dep_id $dep_id" if $dep_id;
- if ($mode eq "list_job") {
- # this is a debugging mode, just print the pstamptool that would have run
- # this is sort of like the mode -noupdate that some other tools support
- print "$command\n";
- } elsif (!$no_update) {
- # mode eq "queue_job"
- my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
- run(command => $command, verbose => $verbose);
- unless ($success) {
- print STDERR @$stderr_buf;
- # XXX TODO: now what? Should we mark the error state for the request?
- # should we keep going for other uris? If so how do we report that some
- # of the work that the request wanted isn't going to get done
- my_die("failed to queue job for request $req_id", $PS_EXIT_UNKNOWN_ERROR);
- }
- } else {
- print "skipping command: $command\n";
- }
- }
- if ( $num_jobs == 0 ) {
- print STDERR "no jobs for row $rownum\n" if $verbose;
- insertFakeJobForRow($row, 1, $PSTAMP_NO_OVERLAP);
- $num_jobs = 1;
- }
- return $num_jobs;
-}
-
-# queue jobs for a collection of request specifications that have the same Images of Interest
-sub queueJobs
-{
- my $mode = shift;
- my $rowList = shift;
- my $imageList = shift;
-
- my $firstRow = $rowList[0];
- my $stage = $firstRow->{IMG_TYPE};
- my $job_type = $firstRow->{JOB_TYPE};
- my $need_magic = $firstRow->{need_magic};
-
- my $num_jobs = 0;
-
- if ($mode eq "list_uri") {
- foreach my $image (@$imageList) {
- print "$image->{image}\n";
- }
- } elsif ($job_type eq "get_image") {
- my $n = scalar @$rowList;
-
- my_die( "error: unexpected number of rows for get_image request: $n", $PS_EXIT_PROG_ERROR) if $n != 1;
-
- $num_jobs = queueGetImageJobs($firstRow, $imageList, $stage, $need_magic, $mode);
-
- } else {
- if (!$imageList or (scalar @$imageList eq 0)) {
- # we didn't find any images for this set of rows. Insert a fake job to carry
- # the status back to the requestor
- foreach my $row (@$rowList) {
- insertFakeJobForRow($row, 1, $PSTAMP_NO_IMAGE_MATCH);
- $num_jobs++;
- }
- return $num_jobs;
- }
-
- my $have_skycells;
- if (($stage eq "raw") or ($stage eq "chip")) {
- $have_skycells = 0;
- } else {
- $have_skycells = 1;
- }
-
- my $thisRun;
-
- my $npoints = 0;
- my ($pointsList, $pointsListName);
- if (scalar @$imageList > 1) {
- ($pointsList, $pointsListName) = tempfile ("/tmp/pointsList.XXXX", UNLINK => !$save_temps);
- foreach my $row (@$rowList) {
- $row->{components} = {};
- if ($row->{skycenter}) {
- print $pointsList "$row->{ROWNUM} $row->{CENTER_X} $row->{CENTER_Y}\n";
- $npoints++;
- } else {
- # this row's center is in pixel coordinates add all images to the component list for this row
- foreach my $i (@$imageList) {
- my $component = $have_skycells ? $i->{skycell_id} : $i->{class_id};
- my_die( "image found with no value for component", $PS_EXIT_UNKNOWN_ERROR)
- if !$component;
- $row->{components}->{$component} = 1;
- }
- }
- }
- close $pointsList;
- } else {
- # only one image. Avoid the expense of dvoImagesAtCoords.
- # queue the job and let ppstamp figure out if the center is valid for the image
- # the resulting result code will be the same.
- foreach my $row (@$rowList) {
- $row->{components} = {};
- my $i = $imageList->[0];
- my $component = $have_skycells ? $i->{skycell_id} : $i->{class_id};
- my_die( "image found with no value for component", $PS_EXIT_UNKNOWN_ERROR) if !$component;
- $row->{components}->{$component} = 1;
- }
- }
-
- my $tess_dir_abs;
- my $last_tess_id = "";
- while ($thisRun = getOneRun($stage, $imageList)) {
- if ($npoints) {
- my_die( "pointsListName is not defined", $PS_EXIT_PROG_ERROR) if !$pointsListName;
- # we collected a set of sky coordinates above.
- # filter the images so that only those that contain the centers of the ROIs are processed
- my $command = "$dvoImagesAtCoords -coords $pointsListName";
- if ($have_skycells) {
- my $tess_id = $thisRun->[0]->{tess_id};
- if ($tess_id ne $last_tess_id) {
- $tess_dir_abs = $ipprc->tessellation_catdir( $tess_id );
- $tess_dir_abs = $ipprc->convert_filename_absolute( $tess_dir_abs );
- $last_tess_id = $tess_id;
- }
- $command .= " -D CATDIR $tess_dir_abs";
- } else {
- my $astrom = $thisRun->[0]->{astrom};
- my_die( "no astrometry file found", $PS_EXIT_UNKNOWN_ERROR) if !$astrom;
- my $astrom_resolved = $ipprc->file_resolve($astrom);
- $command .= " -astrom $astrom_resolved";
- }
- my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
- run(command => $command, verbose => $verbose);
- unless ($success) {
- # don't fail if the program exited normally and exit status was PSTAMP_NO_OVERLAP
- # That just means that the coordinate didn't match any image/skycell
- if (!WIFEXITED($error_code) || (WEXITSTATUS($error_code) ne $PSTAMP_NO_OVERLAP)) {
- print STDERR @$stderr_buf;
- my $rc = WIFEXITED($error_code) ? WEXITSTATUS($error_code) : $PS_EXIT_SYS_ERROR;
- my_die( "dvoImagesAtCoords failed: $rc", $rc);
- }
- }
- # now we have a list of row numbers and components
- # eventually we might want to multiple stamp requests for the same image
- # into the same ppstamp job but not yet. For now we will queue a new
- my @lines = split "\n", join "", @$stdout_buf;
- foreach my $line (@lines) {
- # parse the line, ignoring the ra and dec
- my ($rownum, undef, undef, $component) = split " ", $line;
-
- # I guess since we need this function we should be using a hash for rowList
- my $row = findRow($rownum, $rowList);
- $row->{components}->{$component} = 1;
- }
- }
-
- foreach my $row (@$rowList) {
- $num_jobs += queueJobsForRow($row, $stage, $thisRun, $have_skycells, $need_magic, $mode);
- }
- }
- }
- return $num_jobs;
-}
-
-# $num_jobs = queueGetImageJobs($firstRow, $imageList, $stage, $need_magic, $mode);
-sub queueGetImageJobs
-{
- my $row = shift;
- my $imageList = shift;
- my $stage = shift;
- my $need_magic = shift;
- my $mode = shift;
-
- my $num_jobs = 0;
- my $rownum = $row->{ROWNUM};
- my $option_mask = $row->{OPTION_MASK};
-
- # For dist_bundle we need
- # --camera from $image
- # --stage
- # --stage_id from $image
- # --component from $image
- # --path_base
- # --outdir global to this script
-
- # loop over images
- my $job_num = 0;
- foreach my $image (@$imageList) {
- my $stage_id = $image->{stage_id};
- my $component = $image->{component};
-
- # skip faulted components for now. Should we even be here?
- if ($image->{fault} > 0) {
- printf STDERR "skipping faulted component for $stage $stage_id $component\n" if $verbose;
- next;
- }
-
- $job_num++;
-
- my $imagefile = $image->{image};
- if (($stage ne "stack") and ($need_magic and !$image->{magicked})) {
- # we only get here if req_type is (byid or byexp). For other types the test for magicked is performed
- # in locate_images because it's much more efficient to do the test in the database.
- # For these two modes we fall through to here in order to give feedback to the requestor as
- # to why the request failed to queue jobs.
- print STDERR "skipping non-magicked image $imagefile\n" if $verbose;
- insertFakeJobForRow($row, $job_num, $PSTAMP_NOT_DESTREAKED);
- $num_jobs++;
-
- next;
- }
- my $exp_id = $image->{exp_id};
-
- my $output_base = "$out_dir/${rownum}_${job_num}";
-
- write_params($output_base, $image);
-
- my $newState = "run";
- my $fault = 0;
- my $dep_id;
-
- if ($stage ne 'raw') {
- my $run_state = $image->{state};
- my $data_state = $image->{data_state};
- $data_state = $run_state if $stage eq "stack";
- if (($run_state eq 'goto_purged') or ($data_state eq 'purged') or
- ($run_state eq 'goto_scrubbed') or ($data_state eq 'scrubbed')) {
- # image is gone and it's not coming back
- $newState = 'stop';
- $fault = $PSTAMP_GONE;
- } elsif (($data_state ne 'full') or ($run_state ne 'full' )) {
- # don't wait for update unless the caller asks us to
- if (!($option_mask & $PSTAMP_WAIT_FOR_UPDATE)) {
- $newState = 'stop';
- $fault = $PSTAMP_NOT_AVAILABLE;
- } else {
- # cause the image to be re-made
- # set up to queue an update run
- queue_update_run(\$newState, \$fault, \$dep_id, $image->{image_db},
- $run_state, $stage, $image->{stage_id}, $need_magic);
- }
- }
- }
-
- $num_jobs++;
- my $command = "$pstamptool -addjob -req_id $req_id -job_type $row->{JOB_TYPE}"
- . " -outputBase $output_base -rownum $rownum -state $newState -options $option_mask";
- $command .= " -fault $fault" if $fault;
- $command .= " -exp_id $exp_id" if $exp_id;
- $command .= " -dep_id $dep_id" if $dep_id;
-
- if ($mode eq "list_job") {
- # this is a debugging mode, just print the pstamptool that would have run
- # this is sort of like the mode -noupdate that some other tools support
- print "$command\n";
- } elsif (!$no_update) {
+ if (!$no_update) {
# mode eq "queue_job"
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
@@ -776,9 +784,5 @@
. " -rownum $rownum -state stop -fault $fault";
- if ($mode eq "list_job") {
- # this is a debugging mode, just print the pstamptool that would have run
- # this is sort of like the mode -noupdate that some other tools support
- print "$command\n";
- } elsif (!$no_update) {
+ if (!$no_update) {
# mode eq "queue_job"
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
@@ -796,74 +800,25 @@
}
-sub get_run_id
-{
- my $stage = shift;
- my $image = shift;
-
- if ($stage eq "raw") {
- return $image->{exp_id};
- } elsif ($stage eq "chip") {
- return $image->{chip_id};
- } elsif ($stage eq "warp") {
- return $image->{warp_id};
- } elsif ($stage eq "stack") {
- return $image->{stack_id};
- } elsif ($stage eq "diff") {
- return $image->{diff_id};
- } else {
- my_die("unenexpected stage: $stage found", $PS_EXIT_PROG_ERROR);
- }
-}
-
-# extract components from the imageList that have the same run id and return the list
-sub getOneRun {
- my $stage = shift;
- my $imageList = shift;
-
- # return if array is empty
- return undef if ! @$imageList;
-
- my $last_run_id = 0;
- my @runList;
- while ($imageList->[0]) {
- my $run_id = get_run_id($stage, $imageList->[0]);
-
- last if ($last_run_id and ($run_id ne $last_run_id));
-
- my $image = shift @$imageList;
- $image->{stage} = $stage;
- push @runList, $image;
- $last_run_id = $run_id;
- }
- return \@runList;
-}
-
sub same_images_of_interest {
my $r1 = shift;
my $r2 = shift;
- return 0 if (($r1->{REQ_TYPE} eq "bycoord") or ($r2->{REQ_TYPE} eq "bycoord"));
- return 0 if ($r1->{PROJECT} ne $r2->{PROJECT});
- return 0 if ($r1->{JOB_TYPE} ne $r2->{JOB_TYPE});
+ return 0 if (($r1->{REQ_TYPE} eq "bycoord") or ($r2->{REQ_TYPE} eq "bycoord"));
+ return 0 if (($r1->{JOB_TYPE} eq "get_image") or ($r2->{JOB_TYPE} eq "get_image"));
return 0 if ($r1->{REQ_TYPE} ne $r2->{REQ_TYPE});
return 0 if ($r1->{IMG_TYPE} ne $r2->{IMG_TYPE});
return 0 if ($r1->{ID} ne $r2->{ID});
return 0 if ($r1->{TESS_ID} ne $r2->{TESS_ID});
+ return 0 if ($r1->{COMPONENT} ne $r2->{COMPONENT});
return 0 if ($r1->{REQFILT} ne $r2->{REQFILT});
return 0 if ($r1->{DATA_GROUP} ne $r2->{DATA_GROUP});
return 0 if ($r1->{MJD_MIN} ne $r2->{MJD_MAX});
return 0 if ($r1->{MJD_MAX} ne $r2->{MJD_MAX});
+ return 0 if ($r1->{OPTION_MASK} ne $r2->{OPTION_MASK});
+ return 0 if ($r1->{PROJECT} ne $r2->{PROJECT});
return 0 if ($r1->{inverse} ne $r2->{inverse});
return 0 if ($r1->{unconvolved} ne $r2->{unconvolved});
-
- if (defined($r1->{COMPONENT})) {
- return 0 if !defined $r2->{COMPONENT} or ($r1->{COMPONENT} ne $r2->{COMPONENT});
- } elsif (defined($r2->{COMPONENT})) {
- # if first row has no component all of the images will be retrieved, so
- # the fact that this row has a component is ok. Fall through to return 1
- # XXX Nope this doesn't work. It is consistent with the other logic in some way
- # that i haven't fully thought through
- return 0;
- }
+ # don't combine requests in pixel coordinates
+ return 0 if (($r1->{COORD_MASK} & $PSTAMP_CENTER_IN_PIXELS) || ($r2->{COORD_MASK} & $PSTAMP_CENTER_IN_PIXELS));
return 1;
@@ -911,5 +866,11 @@
sub queue_update_run
{
- my ($r_jobState, $r_fault, $r_dep_id, $imagedb, $state, $stage, $stage_id, $need_magic) = @_;
+ my ($r_jobState, $r_fault, $r_dep_id, $imagedb, $state, $stage, $stage_id, $component, $need_magic) = @_;
+
+ # XXX: The update process for warp and subsequent stages requires
+ # destreaking to run because the -pending queries require it when magicked > 0
+ # queries.
+ # The case of stack-stack diffs is taken care of in pstamp_checkdependent
+ $need_magic = 1 if $imagedb eq 'gpc1';
if (($state ne 'cleaned') and ($state ne 'update') and ($state ne 'goto_cleaned')) {
@@ -918,6 +879,13 @@
my $dep_id;
- my $command = "$pstamptool -getdependent -stage $stage -stage_id $stage_id -imagedb $imagedb";
- $command .= " -no_magic" if !$need_magic;
+ my $command = "$pstamptool -getdependent -stage $stage -stage_id $stage_id -imagedb $imagedb -component $component -outdir $outdir";
+ $command .= " -need_magic" if $need_magic;
+
+ # compute rlabel for the run.
+ # XXX: This bit of policy shouldn't be buried so deeply in the code
+ # For now use one that implies 'postage stamp server' 'update' 'request_label"
+ my $rlabel = "ps_ud_" . $label if $label;
+ $command .= " -rlabel $rlabel" if $rlabel;
+
if (!$no_update) {
my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) =
@@ -929,4 +897,6 @@
chomp $output;
$dep_id = $output;
+ #
+ # XXX: need to fault the request or something
my_die("pstamptool -getdependent returned invalid dep_id", $PS_EXIT_PROG_ERROR) if !$dep_id;
} else {
@@ -936,6 +906,6 @@
$$r_dep_id = $dep_id;
- $r_fault = 0;
- $r_jobState = 'blocked';
+ $$r_fault = 0;
+ $$r_jobState = 'run';
}
@@ -990,4 +960,4 @@
# we don't fault the request here pstamp_parser_run.pl handles that if necessary
- return $fault;
-}
+ exit $fault;
+}
Index: branches/pap/pstamp/scripts/request_finish.pl
===================================================================
--- branches/pap/pstamp/scripts/request_finish.pl (revision 27708)
+++ branches/pap/pstamp/scripts/request_finish.pl (revision 28003)
@@ -21,5 +21,5 @@
use PS::IPP::Config qw( :standard );
-my ( $req_id, $req_name, $req_file, $req_type, $out_dir, $product, $dbname, $dbserver, $verbose, $save_temps, $redirect_output );
+my ( $req_id, $req_name, $req_file, $req_type, $outdir, $product, $dbname, $dbserver, $verbose, $save_temps, $redirect_output );
GetOptions(
@@ -28,5 +28,5 @@
'req_file=s' => \$req_file,
'req_type=s' => \$req_type,
- 'out_dir=s' => \$out_dir,
+ 'outdir=s' => \$outdir,
'product=s' => \$product,
'dbname=s' => \$dbname,
@@ -46,15 +46,15 @@
$err .= "--req_name is required\n" if !$req_name;
$err .= "--product is required\n" if !$product;
-# $err .= "--out_dir is required\n" if !$out_dir;
+# $err .= "--outdir is required\n" if !$outdir;
die "$err" if $err;
-if (!$out_dir) {
- $out_dir = dirname($req_file);
+if (!$outdir) {
+ $outdir = dirname($req_file);
}
if ($redirect_output) {
- my $logDest = "$out_dir/reqfinish.$req_id.log";
+ my $logDest = "$outdir/reqfinish.$req_id.log";
my $ipprc = PS::IPP::Config->new();
$ipprc->redirect_output($logDest);
@@ -79,5 +79,5 @@
}
if ($finish_cmd) {
- my $command = $finish_cmd . " --req_id $req_id --req_name $req_name --req_file $req_file --product $product --out_dir $out_dir";
+ my $command = $finish_cmd . " --req_id $req_id --req_name $req_name --req_file $req_file --product $product --outdir $outdir";
$command .= " --dbname $dbname" if $dbname;
$command .= " --dbserver $dbserver" if $dbserver;
Index: branches/pap/pstamp/src/ppstampMakeStamp.c
===================================================================
--- branches/pap/pstamp/src/ppstampMakeStamp.c (revision 27708)
+++ branches/pap/pstamp/src/ppstampMakeStamp.c (revision 28003)
@@ -36,13 +36,13 @@
double d_before = check->sky->d;
psLogMsg("ppstampMakeStamp", 2, "Before fit Center Pixel RA: %f DEC: %f (degrees)\n",
- r_before * PS_DEG_RAD, d_before * PS_DEG_RAD);
+ r_before * PS_DEG_RAD, d_before * PS_DEG_RAD);
if (outFPA->toSky->type != PS_PROJ_TAN) {
- // we need to make astrometry terms for the output which are centered on the output chip center,
- // but we keep the original plate scale
- psFree(outFPA->toSky);
- outFPA->toSky = psProjectionAlloc (r_before, d_before, inFPA->toSky->Xs, inFPA->toSky->Ys, PS_PROJ_TAN);
-
- if (!pmAstromLinearizeToSky(inFPA, inChip, outFPA, outChip, roi)) {
+ // we need to make astrometry terms for the output which are centered on the output chip center,
+ // but we keep the original plate scale
+ psFree(outFPA->toSky);
+ outFPA->toSky = psProjectionAlloc (r_before, d_before, inFPA->toSky->Xs, inFPA->toSky->Ys, PS_PROJ_TAN);
+
+ if (!pmAstromLinearizeToSky(inFPA, inChip, outFPA, outChip, roi)) {
psFree(check);
psError(PS_ERR_UNKNOWN, false, "Failed to linearize astrometry\n");
@@ -56,7 +56,7 @@
psLogMsg("ppstampMakeStamp", 2, "After fit: Center Pixel RA: %f DEC: %f (degrees)\n",
- r_after * PS_DEG_RAD, d_after * PS_DEG_RAD);
- psLogMsg("ppstampMakeStamp", 2, "Error in fit to astrometry %.2f %.2f (arcseconds)\n",
- (r_after - r_before) * PS_DEG_RAD * 3600, (d_after - d_before) * PS_DEG_RAD * 3600);
+ r_after * PS_DEG_RAD, d_after * PS_DEG_RAD);
+ psLogMsg("ppstampMakeStamp", 2, "Error in fit to astrometry %.2f %.2f (arcseconds)\n",
+ (r_after - r_before) * PS_DEG_RAD * 3600, (d_after - d_before) * PS_DEG_RAD * 3600);
// XXX: should we fail if the fit is bad ??
@@ -238,7 +238,6 @@
return PS_EXIT_DATA_ERROR;
}
- char *fpaName = psMetadataLookupStr(NULL, input->fpa->concepts, "FPA.OBS"); // Name of FPA
pmFPAview *outview = pmFPAviewAlloc(0);
- pmFPAAddSourceFromView(output->fpa, fpaName, outview, output->format);
+ pmFPAAddSourceFromView(output->fpa, outview, output->format);
outview->chip = 0;
Index: branches/pap/pstamp/src/ppstampMosaic.c
===================================================================
--- branches/pap/pstamp/src/ppstampMosaic.c (revision 27708)
+++ branches/pap/pstamp/src/ppstampMosaic.c (revision 28003)
@@ -24,6 +24,5 @@
pmChip *inChip = pmFPAviewThisChip(view, input->fpa);
if (!mChip->hdu && !mChip->parent->hdu) {
- const char *name = psMetadataLookupStr(&status, input->fpa->concepts, "FPA.OBS"); // Name of FPA
- pmFPAAddSourceFromView(mosaic->fpa, name, mosaicView, mosaic->format);
+ pmFPAAddSourceFromView(mosaic->fpa, mosaicView, mosaic->format);
}
psFree(mosaicView);
Index: branches/pap/pstamp/test/detect_query_create
===================================================================
--- branches/pap/pstamp/test/detect_query_create (revision 27708)
+++ branches/pap/pstamp/test/detect_query_create (revision 28003)
@@ -16,4 +16,5 @@
$output, # Name of output table
$query_id,
+ $nostage,
);
@@ -22,4 +23,5 @@
'output|o=s' => \$output,
'query_id|q=s' => \$query_id,
+ 'nostage' => \$nostage,
) or pod2usage( 2 );
@@ -61,4 +63,11 @@
}
];
+unless(defined($nostage)) {
+ push @{ $header }, { name => 'STAGE',
+ writetype => TSTRING,
+ comment => 'processing stage to examine',
+ value => undef
+ };
+}
# Specification of columns to write