Index: /trunk/ippTasks/detrend.assess.pro =================================================================== --- /trunk/ippTasks/detrend.assess.pro (revision 8193) +++ /trunk/ippTasks/detrend.assess.pro (revision 8193) @@ -0,0 +1,110 @@ + +## this file contains the tasks for running the detrend normalization stage + +## these tasks use the queue DetrendRunToAssess + +# select images ready for copy +# new entries are added to queue DetrendRunToAssess +# compare the new list with the ones already selected +task dettool.assess.load + command dettool -stacked -runs + host local + + periods -poll 1 + periods -exec 5 + periods -timeout 30 + + # success + task.exit 0 + # XXX is it necessary for these to be local? + # drop after this task macro is done? + local i Nqueue + + # compare output with newImage queue + # only add entries which don't exist in queue + queuesize stdout -var Nqueue + for i 0 $Nqueue + queuepop stdout -var line + queuepush DetrendRunToAssess -uniq -key 1:2:4 "NEW $line" + end + + # delete existing entries which are DONE + queuedrop DetrendRunToAssess -key 0 DONE + end + + # locked list + task.exit 1 + echo "new.images: exec failure" + end + + # default exit status + task.exit default + echo "new.images: unknown exit status: $EXIT" + end + + # operation times out? + task.exit timeout + echo "new.images: timeout" + end +end + +# copy new images, sending job to desired host +task dettool.assess.process + periods -poll 0.5 + periods -exec 1.0 + periods -timeout 5 + + task.exec + queuesize DetrendRunToAssess -var N + + if ($N == 0) break + if ($network == 0) break + + # look for new images on the internal DetrendRunToAssess queue + # caution with these 'if' statements: syntax errors + # will make the task fail without given a good status + queuepop DetrendRunToAssess -var line -key 0 NEW + if ("$line" == "NULL") break + + strpop line state + queuepush DetrendRunToAssess -replace -key 0 "RUN $line" + + # specify choice of remote host: + # 'anyhost' specified random remote host + # if we needed to specify a host explicitly, use a command like + # one of the following. the -required forces execution on the + # named host, and blocks the task until the host is available + # host kiawe + # host alala -required + host anyhost + + # choose the default recipe appropriate to this type of file + $recipe = `get.recipe $type` + + # create example job options as a demonstration + options "$line" + + # we can put the dettool and ppMerge commands here, or wrap them in a single script + # $urilist = `dettool -stacked -det_id $det_id -iteration $iteration -chip" + # $uri_out = ??? + # command ppNorm $uri_out $uri $uri ... -recipe PPMERGE $recipe ??? + command ASSESS?? $uri_out -det_id $det_id -iteration $iteration -recipe PPMERGE $recipe + end + + # success + task.exit 0 + # XXX need to parse the options to identify the values below (if not part of cmdline?) + exec dettool -addstacked -det_id $det_id -iteration $iteration -class_id $class_id -procrecipe $recipe + queuepush DetrendRunToAssess -replace -key 1:2:4 "DONE $options:0" + end + + # default exit status + task.exit default + queuepush DetrendRunToAssess -replace -key 0 "FAIL $options:0" + end + + # operation times out? + task.exit timeout + queuepush DetrendRunToAssess -replace -key 0 "FAIL $options:0" + end +end Index: /trunk/ippTasks/detrend.mkruns.pro =================================================================== --- /trunk/ippTasks/detrend.mkruns.pro (revision 8193) +++ /trunk/ippTasks/detrend.mkruns.pro (revision 8193) @@ -0,0 +1,95 @@ + +## This script defines tasks which regularly initiate the detrend +## runs. Each detrend type is given its own task, which are run once +## (or twice) per day to build the relevant detrend data + +## when one of these tasks is run, it creates a new detrend run ID in +## the database, and selects the exposures for that run on the basis +## of the specified filters + +# create a new detrend bias-creation run +task mkdetruns.megacam + # command dettool -select -time NOW -24h -type bias -camera megacam + command mkdetruns.megacam + host local + + periods -poll 5 + periods -exec 60 + periods -timeout 30 + trange 18:00 19:00 -nmax 1 + + # success: a new detrend run was created + task.exit 0 + echo "new MEGACAM detrend runs" + queueprint stdout + end + + # default exit status + task.exit default + echo "unable to define a new detrend run" + end + + # operation timed out? is the database down? + task.exit timeout + # send someone email? + echo "unable to define a new detrend run" + end +end + +# create a new detrend bias-creation run +task mkdetruns.isp + # command dettool -select -time NOW -24h -type bias -camera megacam + command mkdetruns.isp + host local + + periods -poll 5 + periods -exec 60 + periods -timeout 30 + trange 18:00 19:00 -nmax 1 + + # success: a new detrend run was created + task.exit 0 + echo "new ISP detrend runs" + queueprint stdout + end + + # default exit status + task.exit default + echo "unable to define a new detrend run" + end + + # operation timed out? is the database down? + task.exit timeout + # send someone email? + echo "unable to define a new detrend run" + end +end + +# create a new detrend bias-creation run +task mkdetruns.gpc + # command dettool -select -time NOW -24h -type bias -camera megacam + command mkdetruns.gpc + host local + + periods -poll 5 + periods -exec 60 + periods -timeout 30 + trange 18:00 19:00 -nmax 1 + + # success: a new detrend run was created + task.exit 0 + echo "new GPC detrend runs" + queueprint stdout + end + + # default exit status + task.exit default + echo "unable to define a new detrend run" + end + + # operation timed out? is the database down? + task.exit timeout + # send someone email? + echo "unable to define a new detrend run" + end +end Index: /trunk/ippTasks/detrend.norm.pro =================================================================== --- /trunk/ippTasks/detrend.norm.pro (revision 8193) +++ /trunk/ippTasks/detrend.norm.pro (revision 8193) @@ -0,0 +1,110 @@ + +## this file contains the tasks for running the detrend normalization stage + +## these tasks use the queue DetrendRunToNormalize + +# select images ready for copy +# new entries are added to queue DetrendRunToNormalize +# compare the new list with the ones already selected +task dettool.norm.load + command dettool -stacked -runs + host local + + periods -poll 1 + periods -exec 5 + periods -timeout 30 + + # success + task.exit 0 + # XXX is it necessary for these to be local? + # drop after this task macro is done? + local i Nqueue + + # compare output with newImage queue + # only add entries which don't exist in queue + queuesize stdout -var Nqueue + for i 0 $Nqueue + queuepop stdout -var line + queuepush DetrendRunToNormalize -uniq -key 1:2:4 "NEW $line" + end + + # delete existing entries which are DONE + queuedrop DetrendRunToNormalize -key 0 DONE + end + + # locked list + task.exit 1 + echo "new.images: exec failure" + end + + # default exit status + task.exit default + echo "new.images: unknown exit status: $EXIT" + end + + # operation times out? + task.exit timeout + echo "new.images: timeout" + end +end + +# copy new images, sending job to desired host +task dettool.norm.process + periods -poll 0.5 + periods -exec 1.0 + periods -timeout 5 + + task.exec + queuesize DetrendRunToNormalize -var N + + if ($N == 0) break + if ($network == 0) break + + # look for new images on the internal DetrendRunToNormalize queue + # caution with these 'if' statements: syntax errors + # will make the task fail without given a good status + queuepop DetrendRunToNormalize -var line -key 0 NEW + if ("$line" == "NULL") break + + strpop line state + queuepush DetrendRunToNormalize -replace -key 0 "RUN $line" + + # specify choice of remote host: + # 'anyhost' specified random remote host + # if we needed to specify a host explicitly, use a command like + # one of the following. the -required forces execution on the + # named host, and blocks the task until the host is available + # host kiawe + # host alala -required + host anyhost + + # choose the default recipe appropriate to this type of file + $recipe = `get.recipe $type` + + # create example job options as a demonstration + options "$line" + + # we can put the dettool and ppMerge commands here, or wrap them in a single script + # $urilist = `dettool -stacked -det_id $det_id -iteration $iteration -chip" + # $uri_out = ??? + # command ppNorm $uri_out $uri $uri ... -recipe PPMERGE $recipe ??? + command ppNorm $uri_out -det_id $det_id -iteration $iteration -recipe PPMERGE $recipe + end + + # success + task.exit 0 + # XXX need to parse the options to identify the values below (if not part of cmdline?) + exec dettool -addstacked -det_id $det_id -iteration $iteration -class_id $class_id -procrecipe $recipe + queuepush DetrendRunToNormalize -replace -key 1:2:4 "DONE $options:0" + end + + # default exit status + task.exit default + queuepush DetrendRunToNormalize -replace -key 0 "FAIL $options:0" + end + + # operation times out? + task.exit timeout + queuepush DetrendRunToNormalize -replace -key 0 "FAIL $options:0" + end +end Index: /trunk/ippTasks/detrend.process.pro =================================================================== --- /trunk/ippTasks/detrend.process.pro (revision 8193) +++ /trunk/ippTasks/detrend.process.pro (revision 8193) @@ -0,0 +1,107 @@ + +## this file contains the tasks for running the detrend processing stage + +## these tasks use the queue DetrendImfilesToProcess +## the DetrendImfilesToProcess queue contains: +## STATE ..... + +# select images ready for copy +# new entries are added to queue DetrendImfilesToProcess +# compare the new list with the ones already selected +task dettool.raw.load + command dettool -raw + host local + + periods -poll 1 + periods -exec 5 + periods -timeout 30 + trange 18:00 23:59 + + # success + task.exit 0 + # XXX is it necessary for these to be local? + # drop after this task macro is done? + local i Nqueue + + # compare output with newImage queue + # only add entries which don't exist in queue + queuesize stdout -var Nqueue + for i 0 $Nqueue + queuepop stdout -var line + queuepush DetrendImfilesToProcess -uniq -key 1:2:4 "NEW $line" + end + + # delete existing entries which are DONE + queuedrop DetrendImfilesToProcess -key 0 DONE + end + + # locked list + task.exit 1 + echo "new.images: exec failure" + end + + # default exit status + task.exit default + echo "new.images: unknown exit status: $EXIT" + end + + # operation times out? + task.exit timeout + echo "new.images: timeout" + end +end + +# copy new images, sending job to desired host +task dettool.raw.process + periods -poll 0.5 + periods -exec 1.0 + periods -timeout 5 + + task.exec + queuesize DetrendImfilesToProcess -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images on the internal DetrendImfilesToProcess queue + # caution with these 'if' statements: syntax errors + # will make the task fail without given a good status + queuepop DetrendImfilesToProcess -var line -key 0 NEW + if ("$line" == "NULL") break + + strpop line state + queuepush DetrendImfilesToProcess -replace -key 0 "RUN $line" + + # specify choice of remote host: + # 'anyhost' specified random remote host + # if we needed to specify a host explicitly, use a command like + # one of the following. the -required forces execution on the + # named host, and blocks the task until the host is available + # host kiawe + # host alala -required + host anyhost + + # choose the default recipe appropriate to this type of file + $recipe = `get.recipe $type` + + # create example job options as a demonstration + options "$line" + command ppImage -file $uri $uri_out -recipe PPIMAGE $recipe + end + + # success + task.exit 0 + # XXX need to parse the options to identify the values below (if not part of cmdline?) + exec dettool -addprocessed -det_id $det_id -exp_id $exp_id -class_id $class_id -uri $output_uri -procrecipe $recipe + queuepush DetrendImfilesToProcess -replace -key 1:2:4 "DONE $options:0" + end + + # default exit status + task.exit default + queuepush DetrendImfilesToProcess -replace -key 0 "FAIL $options:0" + end + + # operation times out? + task.exit timeout + queuepush DetrendImfilesToProcess -replace -key 0 "FAIL $options:0" + end +end Index: /trunk/ippTasks/detrend.resid.pro =================================================================== --- /trunk/ippTasks/detrend.resid.pro (revision 8193) +++ /trunk/ippTasks/detrend.resid.pro (revision 8193) @@ -0,0 +1,111 @@ + +## this file contains the tasks for running the detrend processing stage + +## these tasks use the queue DetrendResidImfiles +## the DetrendResidImfiles queue contains: +## STATE ..... + +# select images ready for copy +# new entries are added to queue DetrendResidImfiles +# compare the new list with the ones already selected +task dettool.resid.load + command dettool -resid + host local + + periods -poll 1 + periods -exec 5 + periods -timeout 30 + trange 18:00 23:59 + + # success + task.exit 0 + # XXX is it necessary for these to be local? + # drop after this task macro is done? + local i Nqueue + + # compare output with newImage queue + # only add entries which don't exist in queue + queuesize stdout -var Nqueue + for i 0 $Nqueue + queuepop stdout -var line + queuepush DetrendResidImfiles -uniq -key 1:2:4 "NEW $line" + end + + # delete existing entries which are DONE + queuedrop DetrendResidImfiles -key 0 DONE + end + + # locked list + task.exit 1 + echo "new.images: exec failure" + end + + # default exit status + task.exit default + echo "new.images: unknown exit status: $EXIT" + end + + # operation times out? + task.exit timeout + echo "new.images: timeout" + end +end + +# copy new images, sending job to desired host +task dettool.resid.process + periods -poll 0.5 + periods -exec 1.0 + periods -timeout 5 + + task.exec + queuesize DetrendResidImfiles -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images on the internal DetrendResidImfiles queue + # caution with these 'if' statements: syntax errors + # will make the task fail without given a good status + queuepop DetrendResidImfiles -var line -key 0 NEW + if ("$line" == "NULL") break + + strpop line state + queuepush DetrendResidImfiles -replace -key 0 "RUN $line" + + # specify choice of remote host: + # 'anyhost' specified random remote host + # if we needed to specify a host explicitly, use a command like + # one of the following. the -required forces execution on the + # named host, and blocks the task until the host is available + # host kiawe + # host alala -required + host anyhost + + # choose the default recipe appropriate to this type of file + $recipe = `get.recipe $type` + + # create example job options as a demonstration + options "$line" + + # need to choose different variations based on the type + # need to supply the matched detrend images... + command ppImage -file $uri $uri_out -recipe PPIMAGE $recipe -bias $master + command ppImage -file $uri $uri_out -recipe PPIMAGE $recipe -flat $master + end + + # success + task.exit 0 + # XXX need to parse the options to identify the values below (if not part of cmdline?) + exec dettool -addprocessed -det_id $det_id -exp_id $exp_id -class_id $class_id -uri $output_uri -procrecipe $recipe + queuepush DetrendResidImfiles -replace -key 1:2:4 "DONE $options:0" + end + + # default exit status + task.exit default + queuepush DetrendResidImfiles -replace -key 0 "FAIL $options:0" + end + + # operation times out? + task.exit timeout + queuepush DetrendResidImfiles -replace -key 0 "FAIL $options:0" + end +end Index: /trunk/ippTasks/detrend.stack.pro =================================================================== --- /trunk/ippTasks/detrend.stack.pro (revision 8193) +++ /trunk/ippTasks/detrend.stack.pro (revision 8193) @@ -0,0 +1,110 @@ + +## this file contains the tasks for running the detrend stacking stage + +## these tasks use the queue DetrendClassIDtoStack + +# select images ready for copy +# new entries are added to queue DetrendClassIDtoStack +# compare the new list with the ones already selected +task dettool.stack.load + command dettool -chip -processed + host local + + periods -poll 1 + periods -exec 5 + periods -timeout 30 + + # success + task.exit 0 + # XXX is it necessary for these to be local? + # drop after this task macro is done? + local i Nqueue + + # compare output with newImage queue + # only add entries which don't exist in queue + queuesize stdout -var Nqueue + for i 0 $Nqueue + queuepop stdout -var line + queuepush DetrendClassIDtoStack -uniq -key 1:2:4 "NEW $line" + end + + # delete existing entries which are DONE + queuedrop DetrendClassIDtoStack -key 0 DONE + end + + # locked list + task.exit 1 + echo "new.images: exec failure" + end + + # default exit status + task.exit default + echo "new.images: unknown exit status: $EXIT" + end + + # operation times out? + task.exit timeout + echo "new.images: timeout" + end +end + +# copy new images, sending job to desired host +task dettool.stack.process + periods -poll 0.5 + periods -exec 1.0 + periods -timeout 5 + + task.exec + queuesize DetrendClassIDtoStack -var N + + if ($N == 0) break + if ($network == 0) break + + # look for new images on the internal DetrendClassIDtoStack queue + # caution with these 'if' statements: syntax errors + # will make the task fail without given a good status + queuepop DetrendClassIDtoStack -var line -key 0 NEW + if ("$line" == "NULL") break + + strpop line state + queuepush DetrendClassIDtoStack -replace -key 0 "RUN $line" + + # specify choice of remote host: + # 'anyhost' specified random remote host + # if we needed to specify a host explicitly, use a command like + # one of the following. the -required forces execution on the + # named host, and blocks the task until the host is available + # host kiawe + # host alala -required + host anyhost + + # choose the default recipe appropriate to this type of file + $recipe = `get.recipe $type` + + # create example job options as a demonstration + options "$line" + + # we can put the dettool and ppMerge commands here, or wrap them in a single script + # $urilist = `dettool -processed -det_id $det_id -iteration $iteration -chip" + # $uri_out = ??? + # command ppMerge $uri_out $uri $uri ... -recipe PPMERGE $recipe + command ppMerge.dettool $uri_out -det_id $det_id -iteration $iteration -recipe PPMERGE $recipe + end + + # success + task.exit 0 + # XXX need to parse the options to identify the values below (if not part of cmdline?) + exec dettool -addstacked -det_id $det_id -iteration $iteration -class_id $class_id -procrecipe $recipe + queuepush DetrendClassIDtoStack -replace -key 1:2:4 "DONE $options:0" + end + + # default exit status + task.exit default + queuepush DetrendClassIDtoStack -replace -key 0 "FAIL $options:0" + end + + # operation times out? + task.exit timeout + queuepush DetrendClassIDtoStack -replace -key 0 "FAIL $options:0" + end +end Index: /trunk/ippTasks/mkdetrend.pro =================================================================== --- /trunk/ippTasks/mkdetrend.pro (revision 8193) +++ /trunk/ippTasks/mkdetrend.pro (revision 8193) @@ -0,0 +1,8 @@ + +input detrend.mkruns.pro +input detrend.process.pro +input detrend.stack.pro +input detrend.norm.pro +input detrend.resid.pro +input detrend.assess.pro + Index: /trunk/ippTasks/notes.txt =================================================================== --- /trunk/ippTasks/notes.txt (revision 8193) +++ /trunk/ippTasks/notes.txt (revision 8193) @@ -0,0 +1,163 @@ + +This directory contains the PanTasks scripts which define the steps +performed by the IPP. These scripts use the ippTools to decide what +images need to be processed, and to update the Metadata Database with +the current state of the analysis. The analysis is performed using +IPP programs which are distributed to the cluster computers as needed. + +1. The Detrend Creation Scripts + +The detrend creation process is among the more complex elements of the +IPP. There are N major steps which must be performed: + +1.1. Define New Detrend Runs + +The script 'detrend.mkruns.pro' defines tasks which initiate new +Detrend Runs. There is a separate task for each camera automatically +processed by the system (mkdetruns.CAMERA). These tasks are executed +on a regular, but infrequent basis. The tasks call external helper +programs which actually run the command dettools -define. The helper +programs are thus responsible for identifying all of the types of +detrend images to be run for a given camera, as well as the correct +set of filters to apply. These tasks are run locally and only affect +the database. PanTasks does not track the detrend runs which are +created. + +- we may need to define these within the PanTasks scripts +- how frequently are these tasks run (1/day?) +- if no raw images are selected by the query, no new detrun should be + created. +- we may need different schedules for the dark (bias/dark) and the + light (flat/fringe) data. And yet another schedule for sky data + (skyflat/skyfringe). + +1.2. Process new raw imfiles + +The script 'detrend.process.pro' defines tasks which processes the new +raw imfiles for any detrend run. This script defines two tasks, +'dettool.raw.load' and 'dettool.raw.process'. The first script +occasionally polls the Metadata Database to select raw imfiles which +have been identified as part of a detrend run. The identified raw +imfiles are added to an internal queue, DetrendImfilesToProcess. Any +raw imfile already in the list is ignored. The imfiles are added to +the queue with a state of NEW. The second task examines the queue for +imfiles with state NEW and defines a ppImage analysis job for that +imfile. The job is sent to pcontrol for processing on the distributed +cluster. At this point, the corresponding queue entry is set to the +state RUN. Upon success, the queue entry is set to DONE and the +database is notified of the successful operation. The DONE entries +are occasionally pruned from the queue; this must be done in the same +task as the initial query and the update of the list based on the +database results. + +- we need to define the columns in the output of dettool -raw +- we need a method to associate an imfile with a node in the cluster +- we need to perform some appropriate action when the queries fail. +- we need a method to generate the output URIs +- we need a method to identify the correct recipe (function of type?) + +1.3. Stack new processed imfiles + +The script 'detrend.stack.pro' defines tasks which stack the newly +processed imfiles for any detrend run iteration. This script defines +two tasks, 'dettool.stack.load and 'dettool.stack.process. The first +script occasionally polls the Metadata Database to select class IDs +for detrend runs for which the input imfiles have all been processed. +The identified detRun/classID combination is added to an internal +queue, DetrendClassIDtoStack. The entries are added to the queue with +a state of NEW. The second task examines the queue for entries with a +state of NEW. It defines a ppMerge analysis job for that +detID/classID. The helper script selects the corresponding input +imfiles and passes them to ppMerge. The job is sent to pcontrol for +processing on the distributed cluster. At this point, the +corresponding queue entry is set to the state RUN. Upon success, the +queue entry is set to DONE and the database is notified of the +successful operation. The DONE entries are occasionally pruned from +the queue; this must be done in the same task as the initial query and +the update of the list based on the database results. + +- we need to define the columns in the output of dettool -chip -processed +- we need a method to associate a classID with a node in the cluster +- we need to perform some appropriate action when the queries fail. +- we need a method to generate the output URIs (not a function of the imfiles...) +- we need a method to identify the correct recipe (function of type?) + + +1.4. Normalize stacked masters + +The script 'detrend.norm.pro' defines tasks which normalize the newly +processed stacks for any detrend run iteration. This script defines +two tasks, 'dettool.norm.load and 'dettool.norm.process. The first +script occasionally polls the Metadata Database to select detrun IDs +for which the classes have all been stacked. The identified +detRunID/iteration combination is added to an internal queue, +DetrendRunToNormalize. The entries are added to the queue with a +state of NEW. The second task examines the queue for entries with a +state of NEW. It defines a ppNorm analysis job for that +detID/iteration. The job is sent to pcontrol for processing on the +distributed cluster. At this point, the corresponding queue entry is +set to the state RUN. Upon success, the queue entry is set to DONE and +the database is notified of the successful operation. The DONE +entries are occasionally pruned from the queue; this must be done in +the same task as the initial query and the update of the list based on +the database results. + +- we need to define the columns in the output of dettool +- we need to perform some appropriate action when the queries fail. +- we need a method to generate the output URIs (not a function of the imfiles...) +- we need a method to identify the correct recipe (function of type?) +- we need to define the ppNorm operation inputs and outputs better + + +1.5. Generate Resid Imfiles + +The script 'detrend.resid.pro' defines tasks which apply the newly +generated detrend image to the corresponding input images to generate +the residual images. This script defines two tasks, +'dettool.resid.load and 'dettool.resid.process. The first script +occasionally polls the Metadata Database to select imfiles for which +the master detrend images has been normalized. The identified imfiles +are added to an internal queue, DetrendResidImfiles. The entries are +added to the queue with a state of NEW. The second task examines the +queue for entries with a state of NEW. It defines a ppImage analysis +job for that imfile. The job is sent to pcontrol for processing on +the distributed cluster. At this point, the corresponding queue entry +is set to the state RUN. Upon success, the queue entry is set to DONE +and the database is notified of the successful operation. The DONE +entries are occasionally pruned from the queue; this must be done in +the same task as the initial query and the update of the list based on +the database results. + +- we need to define the columns in the output of dettool +- can we select the imfiles here without apriori knowledge that the + master has been normalized? +- we need to perform some appropriate action when the queries fail. +- we need a method to generate the output URIs (not a function of the imfiles...) +- we need a method to identify the correct recipe (function of type?) +- we need a method to pass the matching detrend image to ppImage in + the correct context (ie, a bias needs to be supplied to the -bias element...) + +1.6. Assess Detrend Results + +The script 'detrend.assess.pro' defines tasks which assess the +residual statistics for completed detrend runs. This script defines +two tasks, 'dettool.assess.load and 'dettool.assess.process. The +first script occasionally polls the Metadata Database to select +detrunIDs/iterations for which the resid imfiles have all been +generated. The identified detruns are added to an internal queue, +DetrendRunToAssess. The entries are added to the queue with a state +of NEW. The second task examines the queue for entries with a state +of NEW. It defines an analysis job for that collection. The job is +sent to pcontrol for processing on the distributed cluster. At this +point, the corresponding queue entry is set to the state RUN. Upon +success, the queue entry is set to DONE and the database is notified +of the successful operation. The DONE entries are occasionally pruned +from the queue; this must be done in the same task as the initial +query and the update of the list based on the database results. + +- the assess tool needs to trigger a new run, if needed. +- what is the command to run on the resid statistics? +- we need to define the columns in the output of dettool +- can we select the detrend runs here? +- we need to perform some appropriate action when the queries fail. +- we need a method to generate the output URIs (not a function of the imfiles...)