Processing Steps

  • Paul gunzipped and injected all of the 2002 images using the script '' in ipp/DataChallenge.
  • I ran phase0 on all of the 2002 images: pantasks pantasks: input pantasks: init.essence pantasks: input $scripts/ pantasks: run

Note that we currently need to be in the ipp/ippTasks directory to run pantasks (so it knows where to get the scripts).

  • the generic init.controller macro has now be extended a bit. there are now init functions specific to different projects: init.essence, init.isptest. These define the machines and the LOGDIR. However, it is still necessary to be sure your ~/.ipprc file is also pointing at the correct database. For safety, we could add a tests in these macros, but we need to work out the right configuration setup in the longer term.
  • detrun 1 : a simple bias run with only 5 entries. ** Here is the command to initiate the analysis: dettool -pretend -simple -definebyquery -select_exp_type zero \ -det_type bias -select_dateobs_begin 2002-09-05 -select_dateobs_end 2002-09-05T17:38:32.0
    • Here are the pantasks commands.

pantasks pantasks: init.controller pantasks: load.tasks pantasks: run

  • Once pantasks is up and running all tasks, you can run additional bias, flat, etc runs just by calling dettool. The only problem at the moment is the dependencies. None of the pantasks tasks (or the called programs) check to see if biases are available before trying to run the flats, or if flats are available before running the next step. We have some mechanisms define to manage this, by putting blocks on certain steps which are released when the dependecy is done, but this is not yet tested.
  • detruns 2 & 3 : stack all bias images from 2002-09-05, and the range 2002-09-27 : 29 inclusive dettool -pretend -simple -definebyquery -select_exp_type zero -det_type bias -select_dateobs_begin 2002-09-05 -select_dateobs_end 2002-09-06
    • I forget on the first attempt (detrun 2): the object in the detrun must be set to the instrument (CTIO_MOSAIC2). This is an error in the database which cannot be fixed without rebuilding this database. We will wait until after Armin has visited (2006.12.18 - 2006.12.22).
  • detruns 4 & 5 : two flat runs (started at the same time to run in parallel):
    • detrun 4 : V flat for nights 2002.09.28-30
    • detrun 5 : R flat for the same nights
  • detrun 6 : test with 10 biases of the verify mode (success, at least in that it completed)
  • detrun 7 : test with 5 flats of the verify mode for flats.
  • detruns 8 - 10 : verify mode on many biases and V,R flats from 2002. I did not run all of them because we are not yet automatically deleting the intermediate data products, and there is not enough space on that disk to store everything. in fact, these runs included 77 biases, 138 V flats and 153 R flats.
  • detrun 11 : supermacho filter flat run
  • detrun 12 : verify mode on skyflats using master domeflat.

  • list of unique nights in 2002

2002-09-05 2002-09-27 2002-09-28 2002-09-29 2002-10-01 2002-10-02 2002-10-03 2002-10-04 2002-10-05 2002-10-06 2002-10-07 2002-10-08 2002-10-09 2002-10-10 2002-10-11 2002-10-13 2002-10-14 2002-10-29 2002-10-30 2002-10-31 2002-11-02 2002-11-04 2002-11-05 2002-11-06 2002-11-08 2002-11-09 2002-11-10 2002-11-11 2002-11-12 2002-11-13 2002-11-14 2002-11-15 2002-11-27 2002-11-29 2002-12-01 2002-12-02 2002-12-03 2002-12-04 2002-12-05 2002-12-07 2002-12-09 2002-12-13 2002-12-14 2002-12-15 2002-12-16 2002-12-28 2002-12-29 2003-01-02 2003-01-03

problems encountered in the processing

  • some issues I ran into running the flat stacking:
    • I had to modify ppMerge to recognize both skyflat and domeflat types as flat. there are some changes needed in the recipes to handle these variants correctly (see Outstanding Concerns).
    • I was getting many errors for pixels with insufficient valid pixels (probably edge or bad pixels). these were not really a problem, but the error stack filled up and crashed the run. I fixed this by having the appropriate step clear the error stack (it should also check which error occurred; there may be cases where it should bail out).
    • processing speed for ppMerge. The innermost loop in pmReadoutCombine is taking the weight and calculating error = sqrt(weight), passing this into psStats, which then uses weight = error*error. We can re-factor the code so that psStats takes a weight, not an error, and avoid this wasted calculation of the sqrt. I tested the effect on the speed of using the weighted error. I ran tests of both by hand, but this should be an option.
      • without errors: 370s for combination of 5 images
      • with errors: 410 seconds (10% slower) OK, we will use the errors
  • dettool.normstat.process crashed because IPC/ was missing. I installed it by hand, copying from alala. the perl module installation instructions for users really needs to be fixed up. Not only are the instructions for installing those perl packages unclear, the code should not successfully install if required packages are missing. pXtools/scripts needs to check its dependencies.