Index: /branches/czw_branch/cleanup/DataStore/scripts/dsget =================================================================== --- /branches/czw_branch/cleanup/DataStore/scripts/dsget (revision 25149) +++ /branches/czw_branch/cleanup/DataStore/scripts/dsget (revision 25150) @@ -69,5 +69,5 @@ undef $copies; } - if ($copies < 1) { + elsif ($copies < 1) { die "--copies must be >= 1"; } Index: /branches/czw_branch/cleanup/doc/stamps/stampsICD.tex =================================================================== --- /branches/czw_branch/cleanup/doc/stamps/stampsICD.tex (revision 25149) +++ /branches/czw_branch/cleanup/doc/stamps/stampsICD.tex (revision 25150) @@ -76,13 +76,13 @@ \end{table} -The IPP will be configured to periodicly query a set of Data Stores for new request file sets. -We will refer to each Data Store as a \emph{requestor}. -Each request table submitted by a given requstor must have a unique value for -the keyword \code{REQ_NAME}. This value is used for the name of the request's output file set. +The IPP will be configured to periodically query a set of Data Stores for new request file sets. +We will refer to each Data Store as a \emph{requester}. +Each request table submitted must have a unique value for the keyword \code{REQ_NAME}. +This value is used for the name of the request's output file set. The postage stamp server supports three types of "postage stamp" job specified by the value the column \code{JOB_TYPE}. If the value is \emph{stamp} postage stamp images will be extracted from the images of interest. If the value of \code{JOB_TYPE} is \emph{get\_image} a copy -of the selected image(s) with (satelite streaks removed) will be placed on the output file set. +of the selected image(s) with (satellite streaks removed) will be placed on the output file set. Finally if the \code{JOB_TYPE} is \emph{list} the results file will list the input selected images but no postage stamps will be made. @@ -100,11 +100,11 @@ \code{JOB_TYPE} & string & stamp or get\_image & \\ \\ -\code{STAMP_NAME} & string & letters, numbers, '.', '\-' and '\_' & \\ -\code{OPTION_MASK} & U32 & \code{[0-5]} see text & \\ -\\ -\code{REQ_TYPE} & string & bycoord, byexp, or byid & \\ +\code{OPTION_MASK} & U32 & see text & \\ +\\ +\code{REQ_TYPE} & string & bycoord, byexp, byid, bydiff, or byskycell & \\ \code{IMG_TYPE} & string & raw, chip, warp, diff, or stack & \\ -\code{ID} & string & depends on project and REQ\_TYPE& \\ -\code{CLASS_ID} & string & see text & not null for byid raw \& byexp raw \\ +\code{ID} & string & depends on REQ\_TYPE& \\ +\code{TESS_ID} & string & see text & \\ +\code{COMPONENT} & string & see text & \\ \\ \code{COORD_MASK} & U32 & \code{[0-3]} see text & not null if \code{JOB_TYPE} = stamp \\ @@ -130,13 +130,13 @@ \subsubsection{Request Types} The method used to select the images of interest is determined by the value of the REQ\_TYPE column. -There are three alllowed values. +There are three allowed values. \begin{itemize} -\item{\textbf{bycoord}} The center of the region of interest is specifed in celestial coordinates. +\item{\textbf{bycoord}} The center of the region of interest is specified in celestial coordinates. Images are selected by querying the project's DVO database for images with the provided IMG\_TYPE that contain the center. The set of images of interest may be limited by specifying certain criteria as described -in Section \ref{ImageFilters}. +in Section \ref{ImageFilters}. NOTE: this request type is not yet implemented. \item{\textbf{byexp}} The image of interest is the image resulting from a particular exposure @@ -146,7 +146,17 @@ The ID parameter refers to IPP image database key that corresponds to the IMG\_TYPE. e.g: either exp\_id, chip\_id, warp\_id, diff\_id, or stack\_id. + +\item{\textbf{bydiff}} The image with given IMG\_TYPE corresponding to the difference image +whose IMAGEID is equal to ID. This type is useful for requesting stamps based on difference detections. + +\item{\textbf{byskycell}} Images are selected from the skycell with skycell\_id equal to the +value for COMPONENT from the tessellation identified by TESS\_ID. This request type may be +used only with IMG\_TYPE warp, stack, and diff. +The set of images may be limited by filter and date as described in Section \ref{ImageFilters}. +NOTE: this request type is not yet implemented. + \end{itemize} -CLASS\_ID must also be supplied for request specifications where all of the following are true +COMPONENT must also be supplied for request specifications where all of the following are true \begin{itemize} @@ -158,8 +168,7 @@ \item REQ\_TYPE=byexp or REQ\_TYPE=byid -\item IMG\_TYPE=chip or IMG\_TYPE=raw - \end{itemize} +The special value 'all' may be used to select all component images but this is not recommended. \subsubsection{Image Selection Filters}\label{ImageFilters} @@ -169,6 +178,7 @@ The REQFILT parameter may be used to restrict the search for images taken with a given filter. -XXX: What is the namespace for filter names. For example Megacam has filter names like U.9301 which -IPP maps to u. Which name do we allow? Probably should used the mapped names g r i etc. +XXX: What is the name space for filter names. For example Megacam has filter names like U.9301 which +IPP maps to u. Which name do we allow? Perhaps use a LIKE query. + \subsection{The Region of Interest} @@ -177,5 +187,5 @@ a center, width, and height. -The center is defined by the columns CENTER\_X and CENTER\_Y and the dimenions of the rectangle are +The center is defined by the columns CENTER\_X and CENTER\_Y and the dimensions of the rectangle are the values of WIDTH and HEIGHT. The two least signficant bits in value for the column COORD\_MASK determine the type for the coordinates @@ -189,6 +199,6 @@ {\bf COORD\_MASK value} & {\bf Description } \\ \hline -0 & center in RA/DEC; width \& height in arcseconds\\ -1 & center in pixel coordinates; width and height in arcseconds\\ +0 & center in RA/DEC; width \& height in arc seconds\\ +1 & center in pixel coordinates; width and height in arc seconds\\ 2 & center in RA/DEC; width and height in pixels\\ 3 & center x/y, width, and height in pixel coordinates\\ @@ -198,5 +208,5 @@ \end{table} -Another way to describe this is that COORD\_MASK is two bit mask that bitmask that +Another way to describe this is that COORD\_MASK is two bit mask that bit mask that determines the format of the ROI values. In the C language: @@ -212,9 +222,9 @@ When pixel coordinates are used the coordinate system is that of the image. (For chip level images, before -the stamps are created the cells are mosaiced into a single image.) +the stamps are created the cells are mosaicked into a single image.) Any fractional parts in pixel coordinate values are ignored. -The value of \code{OPTION_MASK} is used to select various other options. It is a bitwise or of the following +The value of \code{OPTION\_MASK} is used to select various other options. It is a bitwise or of the following values. @@ -229,10 +239,17 @@ 1 & create postage stamp of the image pixels \\ 2 & create postage stamp of the mask pixels \\ -4 & create postage stamp of the weight pixels \\ -8 & convert WCS to approximate linear transformation in the stamp images\\ -\hline -\end{tabular} -\end{center} -\end{table} +4 & create postage stamp of the variance pixels \\ +1024 & select images based on inverse difference images \\ +\hline +\end{tabular} +\end{center} +\end{table} + +Note that bit zero is currently ignored. A stamp of the image pixels is always included. + +For requests with IMG\_TYPE diff or if \code{REQ_TYPE} is bydiff \code{if (OPTION_MASK & 1024)} and +the corresponding difference run is a warp versus warp diff (bothways) the images +are selected based on the 'inverse' difference images. If this bit is zero the 'positive' +images are selected; @@ -293,5 +310,6 @@ \code{REQ_TYPE} & string & \\ \code{IMG_TYPE} & string & \\ -\code{CLASS_ID} & string & \\ +\code{TESS_ID} & string & \\ +\code{COMPONENT} & string & \\ \\ \code{COORD_MASK} & U32 & \\ @@ -317,24 +335,18 @@ IMG\_NAME give the name of the image in the Data Store file set. -If no STAMP\_NAME is provided in the request specifiction the image file names will be in the form -\code{ROWNUM_I.fits} where I is an integer that ranges from 1 to the number of jobs that the request -specification generated. Mask images will be named \code{ROWNUM_I.mk.fits} and weight images will +The image file names will be in the form \code{ROWNUM_I.fits} where I is an integer that ranges from 1 +to the number of jobs generated for the row's request +specification. + +Mask images will be named \code{ROWNUM_I.mk.fits} and variance images will have the name \code{ROWNUM_I.mk.fits} -If STAMP\_NAME is provided in the resquest specification, the name of the image, mask, and weight files -will be -\begin{itemize} -\item\code{ROWNUM_I_STAMP_NAME.fits} -\item\code{ROWNUM_I_STAMP_NAME.mk.fits}, -\item\code{ROWNUM_I_STAMP_NAME.wt.fits} -\end{itemize} - -JOB\_ID is the value of the postage stamp server's internal job id. This is provided primarily as a debugging -aid. +JOB\_ID is the value of the postage stamp server's internal job id. (This is provided primarily as a debugging +aid.) The column named PROJECT and following columns are copies from the input request specification. -If ERROR\_CODE is non-zero one or more errors occured processing the request specification. -The value of ERROR\_CODE is a bitwise or of the values given in the table\ref{Rerrorcodes}. +If ERROR\_CODE is non-zero an error occurred when processing the request specification or the job. +The values for ERROR\_CODE are listed in table\ref{Rerrorcodes}. @@ -348,20 +360,24 @@ {\bf Name} & {\bf ERROR\_CODE value} & {\bf Description } \\ \hline -NO\_ERROR & 0 & \\ -NO\_MATCHING\_IMAGE & 1 & No image matches the request specification \\ -RA\_DEC\_NOT\_ON\_PIXEL & 2 & The central RA, DEC are on a blind area (for example a inter-chip gap) \\ -RA\_DEC\_NOT\_IN\_FOV & 4 & Wrong position on sky for given MJD \\ -NO\_IMAGE\_FOR\_FILTER & 8 & Image is not in the filter requested \\ -NOT\_AVAILABLE\_PERM & 16 & Image matched but is permanantly unavailable\\ -NOT\_AVAILABLE\_TEMP & 32 & Image matched but is currently unaccessible\\ -INVALID\_REQUEST & 64 & request specification was invalid \\ -SYSTEM\_ERROR & 128 & system error \\ -\hline -\end{tabular} -\end{center} -\end{table} - -In the case of an error parsing the request table a file containing a textual description of the errror -may be added to the resulting file set. +PSTAMP\_SUCCESS & 0 & No errors \\ +PSTAMP\_SYSTEM\_ERROR & 10 & some unspecified system error occurred during processing\\ +PSTAMP\_NOT\_IMPLEMENTED & 11 & Feature not yet implemented \\ +PSTAMP\_UNKNOWN\_ERROR & 12 & Unknown error \\ +PSTAMP\_DUP\_REQUEST & 20 & Request name is a duplicate \\ +PSTAMP\_INVALID\_REQUEST & 21 & Error in a request specification. See parse\_error.txt \\ +PSTAMP\_UNKNOWN\_PRODUCT & 22 & Unknown product in request specification \\ +PSTAMP\_NO\_IMAGE\_MATCH & 23 & No images matched the request \\ +PSTAMP\_NOT\_DESTREAKED & 24 & Image matched, but not yet de-streaked \\ +PSTAMP\_NOT\_AVAILABLE & 25 & Image not available (temporary) \\ +PSTAMP\_GONE & 26 & Image is no longer available \\ +PSTAMP\_NO\_JOBS\_QUEUED & 27 & Request specification yielded no jobs. See parse\_error.txt \\ + +\hline +\end{tabular} +\end{center} +\end{table} + +If parsing of the request specification file results in error output, the contents will be placed +in the file 'parse\_error.txt' in the resulting file set. \end{document} Index: /branches/czw_branch/cleanup/ippScripts/scripts/publish_file.pl =================================================================== --- /branches/czw_branch/cleanup/ippScripts/scripts/publish_file.pl (revision 25149) +++ /branches/czw_branch/cleanup/ippScripts/scripts/publish_file.pl (revision 25150) @@ -39,10 +39,9 @@ # Parse the command-line arguments -my ( $pub_id, $exp_id, $camera, $stage, $stage_id, $format, $product, $workdir ); +my ( $pub_id, $camera, $stage, $stage_id, $format, $product, $workdir ); my ( $dbname, $verbose, $no_update, $save_temps, $redirect ); GetOptions( 'pub_id=s' => \$pub_id, # Publish identifier - 'exp_id=s' => \$exp_id, # Exposure identifier 'camera=s' => \$camera, # Camera name 'stage=s' => \$stage, # Stage of interest @@ -58,8 +57,7 @@ pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; -pod2usage( -msg => "Required options: --pub_id --exp_id --camera --stage --stage_id --product --workdir", +pod2usage( -msg => "Required options: --pub_id --camera --stage --stage_id --product --workdir", -exitval => $PS_EXIT_CONFIG_ERROR) unless defined $pub_id and - defined $exp_id and defined $camera and defined $product and @@ -68,5 +66,5 @@ defined $workdir; -my $outroot = "$workdir/$product.$pub_id.$exp_id"; # Output root name +my $outroot = "$workdir/$product.$pub_id"; # Output root name my $ipprc = PS::IPP::Config->new( $camera ) or @@ -83,4 +81,7 @@ my %files; # Input filenames my %zp; # Zero points +my %exp_id; # Exposure identifiers +my %exp_name; # Exposure names +my %direction; # Direction of subtraction { my $command; # Command to run @@ -114,13 +115,28 @@ if ($stage eq 'diff') { my $skycell_id = $comp->{skycell_id}; + + # Positive direction $files{"$skycell_id.pos"} = $ipprc->filename( "PPSUB.OUTPUT.SOURCES", $path_base ); - $files{"$skycell_id.neg"} = $ipprc->filename( "PPSUB.INVERSE.SOURCES", $path_base ) if - defined $comp->{bothways} and $comp->{bothways}; $zp{"$skycell_id.pos"} = $zp; - $zp{"$skycell_id.neg"} = $zp if defined $comp->{bothways} and $comp->{bothways}; + $exp_id{"$skycell_id.pos"} = $comp->{exp_id_1}; + $exp_name{"$skycell_id.pos"} = $comp->{exp_name_1}; + $direction{"$skycell_id.pos"} = 1; + + # Negative direction + if (defined $comp->{bothways} and $comp->{bothways}) { + $files{"$skycell_id.neg"} = $ipprc->filename( "PPSUB.INVERSE.SOURCES", $path_base ); + $zp{"$skycell_id.neg"} = $zp; + $exp_id{"$skycell_id.neg"} = $comp->{exp_id_2}; + $direction{"$skycell_id.neg"} = 0; + $exp_name{"$skycell_id.neg"} = $comp->{exp_name_2}; + } + } elsif ($stage eq 'camera') { my $cam_id = $comp->{cam_id}; $files{$cam_id} = $ipprc->filename( "PSASTRO.OUTPUT", $path_base ); $zp{$cam_id} = $zp; + $exp_id{$cam_id} = $comp->{exp_id}; + $exp_name{$cam_id} = $comp->{exp_name}; + $direction{$cam_id} = 1; } } @@ -137,4 +153,7 @@ my $zp = $zp{$comp}; + my $exp_id = $exp_id{$comp}; + my $exp_name = $exp_name{$comp}; + my $direction = $direction{$comp}; if ($product eq "IPP-MOPS") { my $outuri = "$outroot.$comp.fits"; @@ -142,5 +161,5 @@ &my_die( "Unable to resolve output file $outuri", $pub_id, $PS_EXIT_SYS_ERROR); - my $command = "$ppMops $file $zp $out"; + my $command = "$ppMops $file $zp $exp_id $exp_name $direction $out"; my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = run(command => $command, verbose => $verbose); Index: /branches/czw_branch/cleanup/ippTasks/dist.pro =================================================================== --- /branches/czw_branch/cleanup/ippTasks/dist.pro (revision 25149) +++ /branches/czw_branch/cleanup/ippTasks/dist.pro (revision 25150) @@ -122,4 +122,5 @@ task.exec + if ($NETWORK == 0) break book npages distToProcess -var N if ($N == 0) @@ -128,6 +129,4 @@ end periods -exec 0.05 - - if ($NETWORK == 0) break # look for new components to process (pantaskState == INIT) Index: /branches/czw_branch/cleanup/ippTasks/publish.pro =================================================================== --- /branches/czw_branch/cleanup/ippTasks/publish.pro (revision 25149) +++ /branches/czw_branch/cleanup/ippTasks/publish.pro (revision 25150) @@ -161,5 +161,4 @@ book getword publishRun $pageName stage -var STAGE book getword publishRun $pageName stage_id -var STAGE_ID - book getword publishRun $pageName exp_id -var EXP_ID book getword publishRun $pageName dbname -var DBNAME @@ -170,5 +169,5 @@ strsub $WORKDIR_TEMPLATE @HOST@ $default_host -var WORKDIR - $run = publish_file.pl --pub_id $PUB_ID --exp_id $EXP_ID --camera $CAMERA --workdir $WORKDIR --product $PRODUCT --stage $STAGE --stage_id $STAGE_ID --redirect-output + $run = publish_file.pl --pub_id $PUB_ID --camera $CAMERA --workdir $WORKDIR --product $PRODUCT --stage $STAGE --stage_id $STAGE_ID --redirect-output add_standard_args run Index: /branches/czw_branch/cleanup/ippTasks/summit.copy.pro =================================================================== --- /branches/czw_branch/cleanup/ippTasks/summit.copy.pro (revision 25149) +++ /branches/czw_branch/cleanup/ippTasks/summit.copy.pro (revision 25150) @@ -449,4 +449,5 @@ end if (("$MD5SUM" != "NULL") && ("$MD5SUM" != "0") && (not($COMPRESS))) +# && (($YEAR > 2008) || (("$YEAR" = "2007") && ($MONTH > 8)))) $run = $run --md5 $MD5SUM end