Commands to run the analysis

  • create the database:

dbadmin project alala ipp gpctest pxadmin -create -dbname gpctest

  • inject the files into the new database

ipp_serial_inject_split.pl --telescope PS1 --camera GPC --workdir /data/ipp000.1/gpc1/20070824 --dbname gpctest c*

  • run registration step in pantasks

module pantasks.pro module register.pro add.database gpc1 controller host add ipp000 controller host add ipp000 run

  • run the first-pass dark creation

dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type dark -select_exp_type dark -workdir file://data/ipp000.0/gpc1/workdir -select_exp_time_min 30 -select_exp_time_max 33 -select_dateobs_begin 2007-12-03T01:23:32 dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type dark -select_exp_type dark -workdir file://data/ipp000.0/gpc1/workdir -select_exp_time_min 60 -select_exp_time_max 65 -select_dateobs_begin 2007-12-03T01:23:32 dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type dark -select_exp_type dark -workdir file://data/ipp000.0/gpc1/workdir -select_exp_time_min 120 -select_exp_time_max 125 -select_dateobs_begin 2007-12-03T01:23:32

module pantasks.pro module detrend.process.pro module detrend.stack.pro module detrend.norm.pro module detrend.resid.pro module detrend.reject.pro controller host add ipp000 controller host add ipp000 controller host add ipp000 controller host add ipp001 controller host add ipp001 controller host add ipp001 run

GPC1 is currently not using a BIAS image -- the OVERSCAN and DARK should take care of it.

  • run the darkmask creation, using the same images that went into each dark creation run. (NOTE: to specify the DARK that is used in the DARKMASK process, you can tweak the constraints in ppImage.config, or just use mySQL to set the other darks to detRunSummary.accept=0, or use dettool -updatedetRun)

dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type darkmask -select_exp_type dark -workdir file://data/ipp000.0/gpc1/workdir -select_exp_time_min 30 -select_exp_time_max 33 -select_dateobs_begin 2007-12-03T01:23:32 dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type darkmask -select_exp_type dark -workdir file://data/ipp000.0/gpc1/workdir -select_exp_time_min 60 -select_exp_time_max 65 -select_dateobs_begin 2007-12-03T01:23:32

job throughput issues

I'm having a bit of concern with the throughput of the registration jobs. The first batch I ran (several hundreds of gpc images), the time for the register.imfile.run step seemed to have quite a large range, up to 160 seconds in some cases, with an average of 16s. I suspect this is the database updates acting like a serial block. I'm doing an experiment now to test the throughput in the absence of db updates. I've commented out the regtool update at the end of register_imfile.pl, and I'm running the jobs on a number of nodes. This will tell me how the throughput is affected by the number of hosts.

database update turned off

  • 1 node : 6.96 : 7.12 : 7.69 (54 jobs)
  • 2 nodes : 6.79 : 7.23 : 8.29 (85 jobs)
  • 6 nodes : 6.97 : 7.81 : 9.78 (57 jobs)
  • 12 nodes : 6.65 : 7.91 : 12.60 (178 jobs)

database update turned on

  • 1 node : 5.10 6.67 10.61 (71 jobs)
  • 2 nodes : 5.07 6.29 8.70 (80 jobs)
  • 6 nodes : 5.69 11.52 16.83 (182 jobs)
  • 12 nodes : 5.58 15.31 30.98 (195 jobs)

the average seems to go up more steeply as we add more nodes that are doing the db update, though it looks like in this run, I'm not getting some of the more extreme outliers on the max time.

So: there is some impact from the db update (the average time goes up somewhat significantly), but there must be other factors causing the significant lurches. One possibility is that in this most recent test, alala (the db server) is not very busy, except for mysql activity. The load is completely dominated by mysql, which also takes up a good chunk of memory. Perhaps yesterday, the load from other sources was higher and it was this contention that was getting in the way of mysql. I know that Mark was running some DVO analyses.

In the end, after running the rest of the data I had:

  • 12 nodes : 4.53 7.45 36.40 (28053 jobs)
  • stats for running a flat set with 9 hosts (ipp000-003)

| alljobs | success | failure |

AV Name | Njobs Tmin Tave Tmax | Njobs Tmin Tave Tmax | Njobs Tmin Tave Tmax | ++ detrend.process.load | 4978 0.25 0.71 44.76 | 4974 0.25 0.71 44.76 | 1 0.26 0.26 0.26 | ++ detrend.process.run | 247 4.04 19.47 140.96 | 213 8.50 22.59 140.96 | 34 4.04 8.01 16.97 | ++ detrend.processexp.load | 4974 0.25 0.71 43.78 | 4971 0.25 0.71 43.78 | 1 0.26 0.26 0.26 | ++ detrend.processexp.run | 4 14.76 22.78 31.46 | 4 14.76 22.78 31.46 | 0 NONE 0.00 0.00 | ++ detrend.stack.load | 4979 0.25 0.73 43.77 | 4975 0.25 0.73 43.77 | 1 0.77 0.77 0.77 | ++ detrend.stack.run | 180 505.58 818.80 2513.38 | 180 505.58 897.39 2513.38 | 0 NONE 0.00 0.00 | ++ detrend.normstat.load | 4982 0.25 0.73 42.91 | 4981 0.25 0.73 42.91 | 0 NONE 0.00 0.00 | ++ detrend.normstat.run | 3 27.79 29.99 32.85 | 3 27.79 29.99 32.85 | 0 NONE 0.00 0.00 | ++ detrend.norm.load | 4977 0.25 0.72 42.91 | 4974 0.25 0.72 42.91 | 2 0.51 0.84 1.16 | ++ detrend.norm.run | 127 9.21 21.65 50.46 | 127 9.21 21.65 50.46 | 0 NONE 0.00 0.00 | ++ detrend.normexp.load | 4977 0.25 0.69 42.91 | 4974 0.25 0.69 42.91 | 2 0.26 0.38 0.51 | ++ detrend.normexp.run | 3 2.64 3.11 4.00 | 3 2.64 3.11 4.00 | 0 NONE 0.00 0.00 | ++ detrend.resid.load | 4984 0.25 0.66 42.91 | 4983 0.25 0.66 42.91 | 0 NONE 0.00 0.00 | ++ detrend.resid.run | 2520 11.98 29.31 79.63 | 2502 11.98 29.53 79.63 | 2 13.49 21.73 29.96 | ++ detrend.residexp.load | 4969 0.25 0.72 42.91 | 4967 0.25 0.72 42.91 | 1 0.26 0.26 0.26 | ++ detrend.residexp.run | 40 19.63 24.51 41.92 | 40 19.63 31.43 41.92 | 0 NONE 0.00 0.00 | ++ detrend.reject.load | 4967 0.25 0.66 42.91 | 4966 0.25 0.66 42.91 | 0 NONE 0.00 0.00 | ++ detrend.reject.run | 2 5.93 6.18 6.42 | 2 5.93 6.18 6.42 | 0 NONE 0.00 0.00 |

  • stats after running all night (no convergence on the flat: flip-flopping...)

| alljobs | success | failure |

AV Name | Njobs Tmin Tave Tmax | Njobs Tmin Tave Tmax | Njobs Tmin Tave Tmax | ++ detrend.process.load | 14778 0.25 0.73 44.76 | 14774 0.25 0.73 44.76 | 1 0.26 0.26 0.26 | ++ detrend.process.run | 247 4.04 19.47 140.96 | 213 8.50 22.59 140.96 | 34 4.04 8.01 16.97 | ++ detrend.processexp.load | 14772 0.25 0.72 43.78 | 14768 0.25 0.72 43.78 | 2 0.26 0.38 0.51 | ++ detrend.processexp.run | 4 14.76 22.78 31.46 | 4 14.76 22.78 31.46 | 0 NONE 0.00 0.00 | ++ detrend.stack.load | 14779 0.25 0.75 43.77 | 14774 0.25 0.75 43.77 | 2 0.77 0.86 0.95 | ++ detrend.stack.run | 480 505.58 809.25 2513.38 | 480 505.58 838.67 2513.38 | 0 NONE 0.00 0.00 | ++ detrend.normstat.load | 14777 0.25 0.74 42.91 | 14775 0.25 0.74 42.91 | 1 0.51 0.51 0.51 | ++ detrend.normstat.run | 8 27.79 30.89 35.70 | 8 27.79 30.89 35.70 | 0 NONE 0.00 0.00 | ++ detrend.norm.load | 14771 0.25 0.73 42.91 | 14767 0.25 0.73 42.91 | 3 0.25 0.64 1.16 | ++ detrend.norm.run | 475 9.21 21.97 50.46 | 475 9.21 21.97 50.46 | 0 NONE 0.00 0.00 | ++ detrend.normexp.load | 14773 0.25 0.70 42.91 | 14769 0.25 0.70 42.91 | 3 0.25 0.34 0.51 | ++ detrend.normexp.run | 8 2.28 3.06 4.00 | 8 2.28 3.06 4.00 | 0 NONE 0.00 0.00 | ++ detrend.resid.load | 14792 0.25 0.66 42.91 | 14789 0.25 0.66 42.91 | 2 0.26 0.38 0.50 | ++ detrend.resid.run | 9405 1.41 29.52 98.26 | 9398 11.98 29.58 98.26 | 7 1.41 16.25 32.62 | ++ detrend.residexp.load | 14720 0.25 0.72 42.91 | 14718 0.25 0.72 42.91 | 1 0.26 0.26 0.26 | ++ detrend.residexp.run | 140 18.27 27.57 41.92 | 140 18.27 29.84 41.92 | 0 NONE 0.00 0.00 | ++ detrend.reject.load | 14718 0.25 0.68 42.91 | 14716 0.25 0.68 42.91 | 1 0.51 0.51 0.51 | ++ detrend.reject.run | 7 5.54 6.43 9.83 | 7 5.54 6.43 9.83 | 0 NONE 0.00 0.00 |

  • stats after running flats all night

| alljobs | success | failure |

AV Name | Njobs Tmin Tave Tmax | Njobs Tmin Tave Tmax | Njobs Tmin Tave Tmax | ++ detrend.process.load | 16911 0.25 0.60 17.40 | 16911 0.25 0.60 17.40 | 0 NONE 0.00 0.00 | ++ detrend.process.run | 1785 4.08 22.93 80.09 | 1711 11.15 23.89 80.09 | 74 4.08 6.00 11.36 | ++ detrend.processexp.load | 16898 0.25 0.59 14.11 | 16898 0.25 0.59 14.11 | 0 NONE 0.00 0.00 | ++ detrend.processexp.run | 29 15.33 22.42 36.09 | 29 15.33 22.42 36.09 | 0 NONE 0.00 0.00 | ++ detrend.stack.load | 16890 0.25 0.60 14.10 | 16889 0.25 0.60 14.10 | 1 0.51 0.51 0.51 | ++ detrend.stack.run | 240 80.14 182.13 470.44 | 240 80.14 195.62 470.44 | 0 NONE 0.00 0.00 | ++ detrend.normstat.load | 16880 0.25 0.60 17.43 | 16880 0.25 0.60 17.43 | 0 NONE 0.00 0.00 | ++ detrend.normstat.run | 4 18.69 20.21 23.86 | 4 18.69 20.21 23.86 | 0 NONE 0.00 0.00 | ++ detrend.norm.load | 16890 0.25 0.60 17.43 | 16890 0.25 0.60 17.43 | 0 NONE 0.00 0.00 | ++ detrend.norm.run | 240 8.40 21.40 71.10 | 240 8.40 22.99 71.10 | 0 NONE 0.00 0.00 | ++ detrend.normexp.load | 16883 0.25 0.63 17.42 | 16883 0.25 0.63 17.42 | 0 NONE 0.00 0.00 | ++ detrend.normexp.run | 3748 5.26 6.72 33.12 | 3 7.09 11.04 13.33 | 3745 5.26 6.72 33.12 | ++ detrend.resid.load | 16892 0.25 0.59 17.42 | 16892 0.25 0.59 17.42 | 0 NONE 0.00 0.00 | ++ detrend.resid.run | 1801 11.87 29.51 78.85 | 1800 11.87 29.80 78.85 | 1 15.87 15.87 15.87 | ++ detrend.residexp.load | 16853 0.25 0.60 17.42 | 16853 0.25 0.60 17.42 | 0 NONE 0.00 0.00 | ++ detrend.residexp.run | 30 15.26 23.59 42.00 | 30 15.26 27.88 42.00 | 0 NONE 0.00 0.00 | ++ detrend.reject.load | 16862 0.25 0.59 17.42 | 16861 0.25 0.59 17.42 | 1 5.30 5.30 5.30 | ++ detrend.reject.run | 4 2.64 4.93 10.44 | 4 2.64 4.93 10.44 | 0 NONE 0.00 0.00 |

  • other dettool commands: dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type flatmask -select_exp_type skyflat -workdir file://data/ipp000.0/gpc1/workdir -select_filter i.00000 -select_sat_pixel_frac_max 0.2 -select_exp_time_max 29.5 dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type dark -select_exp_type dark -workdir file://data/ipp000.0/gpc1/workdir -select_exp_time_min 60 -select_exp_time_max 65 -select_dateobs_begin 2007-12-02T22:00:00 -select_dateobs_end 2007-12-03T01:00:00 dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type dark -select_exp_type dark -workdir file://data/ipp000.0/gpc1/workdir -select_exp_time_min 110 -select_exp_time_max 130 -select_dateobs_begin 2007-12-02T22:00:00 -select_dateobs_end 2007-12-03T01:00:00 dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type flatmask -select_exp_type skyflat -workdir file://data/ipp000.0/gpc1/workdir -select_filter z.00000 -select_sat_pixel_frac_max 0.2 -select_exp_time_max 29.5 dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type darkmask -select_exp_type dark -workdir file://data/ipp000.0/gpc1/workdir -select_exp_time_min 60 -select_exp_time_max 65 -select_dateobs_begin 2007-12-02T22:00:00 -select_dateobs_end 2007-12-03T01:00:00 dettool -simple -dbname gpc1 -definebyquery -inst GPC1 -det_type darkmask -select_exp_type dark -workdir file://data/ipp000.0/gpc1/workdir -select_exp_time_min 110 -select_exp_time_max 130 -select_dateobs_begin 2007-12-02T22:00:00 -select_dateobs_end 2007-12-03T01:00:00
  • tessalations for PR images: skycells -D CATDIR skycells.4n.M31 4 -fix-ns -euler 1.5 1.8 -nx 6 -ny 6 -scale 0.333 skycells -D CATDIR skycells.4n.Holmes 4 -fix-ns -euler -0.1 -0.5 -nx 6 -ny 6 -scale 0.333
  • running the warps: warptool -definerun -mode warp -workdir /data/ipp002.0/GPC1/prpics warptool -addinputexp -warp_id 1 -cam_id 82 warptool -updaterun -warp_id 1 -state run warptool -runone -mode warp -workdir /data/ipp002.0/GPC1/prpics -cam_id 82

note that i need to change the tessalation entry in gpc1/camera.config between the Holmes and M31 runs

(I had to clear the camProcessedExp faults manually - they are from the astrometry problems?)

  • running the stacks stacktool -definerun -workdir /data/ipp002.0/GPC1/prpics -skycell_id skycell.0122.00 -tess_id HOLMES -warp_id 2 -warp_id 3 -warp_id 4 -warp_id 5
  • I manually, in mana, adjusted the gain for a number of g and r chips. I am rerunning the warps now.
  • I was able to fix the astrometry problems for Orion by fixing the "psastroFixChips" function. At least, this now works for a per-chip astrometry model; i don't have a good gpc1 mosaic astrometry model to test that version. The Orion 15s image data is now processing (1/11/08 afternoon) in single-chip mode. when it is done, I can run astrometry with fix chips turned on for the short exposures: set PSASTRO.FIX.CHIPS to true in psastro.config

psastro -file "*.smf" output