| | 1 | == Restarting the Fall 2006 / Jan 2007 ISP Data == |
| | 2 | |
| | 3 | There were still some header errors for the ISP data (filter, cd matrix, etc). Erik and Josh worked to fix this. I am restarting the full analysis to have a complete example. |
| | 4 | |
| | 5 | * create the database: |
| | 6 | |
| | 7 | dbadmin project alala ipp isp_v2 |
| | 8 | pxadmin -create -dbname isp_v2 |
| | 9 | |
| | 10 | * inject the files into the new database |
| | 11 | |
| | 12 | foreach dir (oi*) |
| | 13 | ipp_serial_inject.pl $dir/*.fits --workdir path://ISP_V2/$dir --dbname isp_v2 |
| | 14 | echo $dir |
| | 15 | end |
| | 16 | |
| | 17 | * register the images (in pantasks) |
| | 18 | |
| | 19 | module pantasks.pro |
| | 20 | module.tasks |
| | 21 | input cluster.pro |
| | 22 | init.cluster |
| | 23 | list DB -add isp_v2 |
| | 24 | |
| | 25 | * bias |
| | 26 | |
| | 27 | dettool -pretend -filelevel isp -simple -definebyquery -inst ISP-Apogee -det_type bias -select_exp_type bias -dbname isp_v2 -workdir path://ISP_V2/detrend/bias -select_dateobs_begin 2007-01-13T00:00:00 |
| | 28 | |
| | 29 | |
| | 30 | * build the dark frames |
| | 31 | |
| | 32 | dettool -simple -filelevel isp -definebyquery -inst ISP-Apogee -det_type dark -select_exp_type dark -dbname isp_v2 -workdir path://ISP_V2/detrend/dark -select_dateobs_begin 2007-01-14T00:00:00 -exp_time_min 10 -exp_time_max 60 -select_exp_time_min 10 |
| | 33 | |
| | 34 | dettool -simple -filelevel isp -definebyquery -inst ISP-Apogee -det_type dark -select_exp_type dark -dbname isp_v2 -workdir path://ISP_V2/detrend/dark -select_dateobs_begin 2007-01-14T00:00:00 -exp_time_max 10 -select_exp_time_max 10 |
| | 35 | |
| | 36 | * shutter |
| | 37 | |
| | 38 | I first tried the following line. This included several very short exposures which are probably uncorrectable (exp_time < min_dt). the analysis did not automatically reject them. this is partly because the rejections are not measuring a sufficiently discriminating criterion and partly because the 'residuals' still include the flat term (only shutter is corrected). the first motivates the measurement of a gridded or binned stdev statistic (eg, stdev of the 16x16 binned image); the second motivates a modification to the analysis so that the residual generation applies both shutter and flat-field (and also that the shutter image includes the slope and intercept values). |
| | 39 | |
| | 40 | dettool -simple -filelevel isp -definebyquery -inst ISP-Apogee -det_type shutter -select_exp_type skyflat -select_filter r -dbname isp_v2 -workdir path://ISP_V2/detrend/shutter -select_dateobs_begin 2007-01-14T04:00:00 -select_dateobs_end 2007-01-14T05:00:00 |
| | 41 | |
| | 42 | for the short term, I re-ran the shutter correction analysis, excluding the shortest exposures (exp_time < 0.05). i tried the two sets below (g from sunset and sunrise). both yielded similar results, though the first was a little bit better. |
| | 43 | |
| | 44 | dettool -pretend -simple -definebyquery -inst ISP-Apogee -det_type shutter -select_exp_type skyflat -select_filter g -dbname isp_v2 -workdir path://ISP_V2/detrend/shutter -select_dateobs_begin 2007-01-14T04:00:00 -select_dateobs_end 2007-01-14T05:00:00 -select_exp_time_min 0.04 |
| | 45 | |
| | 46 | dettool -pretend -simple -definebyquery -inst ISP-Apogee -det_type shutter -select_exp_type skyflat -select_filter g -dbname isp_v2 -workdir path://ISP_V2/detrend/shutter -select_dateobs_begin 2007-01-14T16:00:00 -select_dateobs_end 2007-01-14T17:00:00 -select_exp_time_min 0.05 |
| | 47 | |
| | 48 | * flat |
| | 49 | |
| | 50 | I built g,r,i,z,y flats at the same time, running the following commands: |
| | 51 | |
| | 52 | exec dettool -simple -select_exp_time_min 0.05 -definebyquery -inst ISP-Apogee -det_type flat -select_exp_type skyflat -dbname isp_v2 -workdir path://ISP_V2/detrend/flat/g -select_filter g -filter g -select_dateobs_begin 2007-01-14T00:00:00 -select_dateobs_end 2007-01-15T00:00:00 |
| | 53 | |
| | 54 | exec dettool -simple -select_exp_time_min 0.05 -definebyquery -inst ISP-Apogee -det_type flat -select_exp_type skyflat -dbname isp_v2 -workdir path://ISP_V2/detrend/flat/r -select_filter r -filter r -select_dateobs_begin 2007-01-14T00:00:00 -select_dateobs_end 2007-01-15T00:00:00 |
| | 55 | |
| | 56 | exec dettool -simple -select_exp_time_min 0.05 -definebyquery -inst ISP-Apogee -det_type flat -select_exp_type skyflat -dbname isp_v2 -workdir path://ISP_V2/detrend/flat/i -select_filter i -filter i -select_dateobs_begin 2007-01-14T00:00:00 -select_dateobs_end 2007-01-15T00:00:00 |
| | 57 | |
| | 58 | exec dettool -simple -select_exp_time_min 0.05 -definebyquery -inst ISP-Apogee -det_type flat -select_exp_type skyflat -dbname isp_v2 -workdir path://ISP_V2/detrend/flat/z -select_filter z -filter z -select_dateobs_begin 2007-01-14T00:00:00 -select_dateobs_end 2007-01-15T00:00:00 |
| | 59 | |
| | 60 | exec dettool -simple -select_exp_time_min 0.05 -definebyquery -inst ISP-Apogee -det_type flat -select_exp_type skyflat -dbname isp_v2 -workdir path://ISP_V2/detrend/flat/y -select_filter y -filter y -select_dateobs_begin 2007-01-14T00:00:00 -select_dateobs_end 2007-01-15T00:00:00 |
| | 61 | |
| | 62 | |
| | 63 | == Running the Fall 2006 / Jan 2007 ISP Data == |
| | 64 | 2007.05.24 : EAM |
| | 65 | |
| | 66 | I am running the ISP data from the Fall 2006 and Jan 2007 through the IPP using the 2.1 pre-release code base. These are my notes on the process and issues I have encountered. I am running the analysis as the user ipp, referring to the psconfig ipp-2.1pre installation. I am using a cvs checkout under my own user ID to install this tree, so I can make fixes to the code as I encounter problems. |
| | 67 | |
| | 68 | I am using a local file (cluster.pro) to define additional macros. for example, |
| | 69 | |
| | 70 | module pantasks.pro |
| | 71 | module.tasks |
| | 72 | input cluster.pro |
| | 73 | init.cluster |
| | 74 | list DB -add isp |
| | 75 | |
| | 76 | |
| | 77 | * registration : I ran into some trouble because ipp000 did not have the perl modules installed for ipp-2.1pre. I was fooled into thinking the installation was OK because my PERL5LIB when I built the tree included an entry to my own psconfig installation, where the modules were installed. Since the ipp user does not include that path, programs running as that user on ipp000 failed. To make it easier to track this down (which node had the problem?), I added the JOB_HOSTNAME to the pantasks internal variables, and echo it on job failures. |
| | 78 | |
| | 79 | * the URIs displayed by ippMonitor have the full path, not path://ISP/foo. but, the db queries seem to show the path:// form? |
| | 80 | |
| | 81 | * bias : I started a bias run using the images from Jan only: |
| | 82 | |
| | 83 | dettool -simple -definebyquery -inst ISP-Apogee -det_type bias -select_exp_type bias -dbname isp -workdir path://ISP/detrend/bias -select_dateobs_begin 2007-01-13T00:00:00 |
| | 84 | |
| | 85 | I ran into trouble with the isp/format.config: CCDBIN1,2 was not set for all images; XBIN,YBIN were set. '''These images should be fixed to reflect the ICD: oi3986, oi4035 - oi4114, oi4117, oi4118.''' |
| | 86 | |
| | 87 | |
| | 88 | |
| | 89 | steps to run flats: |
| | 90 | |
| | 91 | dettool -pretend -definebyquery -inst ISP-Apogee -filter r -det_type flat -select_exp_type skyflat -select_filter r -workdir path://ISP_PO5/2007.01/mkflat.r.0 -dbname isp |
