Changes between Version 31 and Version 32 of LAPintoPSPS
- Timestamp:
- Mar 1, 2013, 3:19:23 PM (13 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
LAPintoPSPS
v31 v32 26 26 27 27 * once queued this is straightforward. Some tweaks may include to build the minidvodbs around the cluster (not just ipp004/ipp005) 28 * see progress on either heather's addstar wiki (link) or from ippmonitor 28 29 29 30 == Merge dvodb into big dvodb == 30 31 31 * once the slices are built, we merge into a big parallel dvodb, and run the resort/averages/relphot commands on the slice in question or around the whole thing? 32 {{{ 32 * we do not merge while ipptopsps is running. Once a slice of sky has been ingested and once ipptopsps stops, then we merge more slices together into the big one. 33 34 * once the slices are merged into a big parallel dvodb, run the resort/averages/relphot commands on the big dvodb 35 {{{ 36 addstar -resort (stuff) 33 37 setphot -D CATDIR $catdir -update -ubercal -no-metadata ~ipp/ucalqw_noref.fits 34 38 relphot -D CATDIR $catdir -statmode WT_MEAN -max-density 2000 -basic-image-search -update g,r,i,z,y -nloop 16 -boundary-tree ~ipp/RINGS.V3.tree.v2.fits -D CAMERA gpc1 35 39 relastro -D CATDIR $catdir -update-objects -reset -update -D CAMERA gpc1 -D CATDIR $catdir 36 40 }}} 41 37 42 38 43 == IPPtoPSPS creates batches based on slices == … … 43 48 * dec: -33 to 75 deg 44 49 * this is a combination of slice16 and slice18 with a 3 degree border, and i propose that we be very conservative with the cap. 45 * task: need ipptopsps to handle parallel dvodb 50 * task: need ways to monitor this 51 46 52 == PSPS loads batches == 47 53 … … 51 57 == Slices == 52 58 53 These are the following slices, labeled by Heather. The goal is to load connected slices (ie if slice18 is first then load either slice20 or slice 16 next). This was last updated 1-22. 54 55 ||slice name || load order || ra || dec || ra || dec || ipptopsps ranges ra || ipptopsps ranges dec || cam || skycal || merged || relphoted/etc ||comments || 56 || slice0 || || 0 to 2hr || -30 to 90 deg || 0 to 30 deg || -30 to 90 deg || 3 to 27 deg || -33 to 75 deg || || || || || || 57 || slice2 || || 2 to 4hr || -30 to 90 deg || 30 to 60 deg || -30 to 90 deg || 33 to 57 deg || -33 to 75 deg || || || || || || 58 || slice4 || || 4 to 6hr || -30 to 90 deg || 60 to 90 deg || -30 to 90 deg || 63 to 87 deg || -33 to 75 deg || || || || || || 59 || slice6 || || 6 to 8hr || -30 to 90 deg || 90 to 120 deg || -30 to 90 deg || 93 to 117 deg || -33 to 75 deg || || || || || || 60 || slice8 || 8* || 8 to 10hr || 20 to 90 deg || 120 to 150 deg || -30 to 90 deg || 123 to 147 deg || -33 to 75 deg || in process|| || || || || 61 || slice8.2 || 7* || 8 to 10hr || -30 to 20 deg || 120 to 150 deg || -30 to 90 deg || 123 to 147 deg || -33 to 75 deg || done || || || || || 62 || slice10 || || 10 to 12hr || 20 to 90 deg || 150 to 180 deg || -30 to 90 deg || 153 to 177 deg || -33 to 75 deg || || || || || || 63 || slice10.2 || 6* || 10 to 12hr || -30 to 20 deg || 150 to 180 deg || -30 to 90 deg || 153 to 177 deg || -33 to 75 deg |||| || || || || 64 || slice12 || || 12 to 14hr || 20 to 90 deg || 180 to 210 deg || -30 to 90 deg || 183 to 207 deg || -33 to 75 deg |||| || || || || 65 || slice12.2 || 5*|| 12 to 14hr || -30 to 20 deg || 180 to 210 deg || -30 to 90 deg || 183 to 207 deg || -33 to 75 deg |||| || || || || 66 || slice14 || || 14 to 16hr || 20 to 90 deg || 210 to 240 deg || -30 to 90 deg || 213 to 237 deg || -33 to 75 deg |||| || || || || 67 || slice14.2 || 4* || 14 to 16hr || -30 to 20 deg || 210 to 240 deg || -30 to 90 deg || 213 to 237 deg || -33 to 75 deg |||| || || || || 68 || slice16 || || 16 to 18hr || 20 to 90 deg || 240 to 270 deg || 20 to 90 deg || 243 to 267 deg || 23 to 75 deg |||| || || || || 69 || slice16.2 || 3* || 16 to 18hr || -30 to 20 deg || 240 to 270 deg || -30 to 20 deg || 243 to 267 deg || -33 to 17 deg || in process|| || || || || 70 || slice18 || 1 || 18 to 20hr || 20 to 90 deg || 270 to 300 deg || 20 to 90 deg || 273 to 297 deg || 23 to 75 deg ||done || || || || (this is kepler, and below is the galactic center)|| 71 || slice18.2 || 2* || 18 to 20hr || -30 to 20 deg || 270 to 300 deg || -30 to 20 deg || 273 to 297 deg || -33 to 17 deg ||done || || || || (worry about the overlap between the 2 slice18s) || 72 || slice20 || || 20 to 22hr || -30 to 90 deg || 300 to 330 deg || -30 to 90 deg || 303 to 327 deg || -33 to 75 deg |||| || || || || 73 || slice22 || ||22 to 24hr || -30 to 90 deg || 330 to 360 deg || -30 to 90 deg || 333 to 357 deg || -33 to 75 deg |||| || || || || 74 || slice.top || || || || || || || |||| || || || || 59 These are the following slices, labeled by Heather. The goal is to load connected slices (ie if slice18 is first then load either slice20 or slice 16 next). This was last updated 3-01. 60 61 ||slice name || ipptopsps load order || ra || dec || cam || skycal || merged cam || merged skycal || relphoted/etc || ingested by ipptopsps||comments || 62 || slice0 || || 0 to 2hr || 20 to 75 deg || || || || || || || 63 || slice0 || || 0 to 2hr || -30 to 20 deg || || || || || || || 64 || slice2 || || 2 to 4hr || 20 to 75 deg || || || || || || || 65 || slice2 || || 2 to 4hr || -30 to 20 deg || || || || || || || 66 || slice4 || || 4 to 6hr || 20 to 75 deg || || || || || || || 67 || slice4 || || 4 to 6hr || -30 to 20 deg || || || || || || || 68 || slice6 || || 6 to 8hr || 20 to 75 deg || || || || || || || 69 || slice6 || || 6 to 8hr || -30 to 20 deg || || || || || || || 70 || slice8 || || 8 to 10hr || 20 to 75 deg || partial || partial || || || || || 71 || slice8.2 || || 8 to 10hr || -30 to 20 deg || partial || partial || || || || || 72 || slice10 || 2 * || 10 to 12hr || 20 to 75 deg || done || done || *next || *next || || || 73 || slice10.2 || 3 * || 10 to 12hr || -30 to 20 deg || done || done || *next || *next || || || 74 || slice12 || 4 * || 12 to 14hr || 20 to 75 deg || done || done || *next || *next || || || 75 || slice12.2 || 5 * || 12 to 14hr || -30 to 20 deg || done || done || *next || *next || || || 76 || slice14 || || 14 to 16hr || 20 to 75 deg || done || in progress || *next || || || || 77 || slice14.2 || || 14 to 16hr || -30 to 20 deg || done || in progress || *next || || || || 78 || slice16 || || 16 to 18hr || 20 to 75 deg || done || in progress || *next || || || || 79 || slice16.2 || || 16 to 18hr || -30 to 20 deg || done || in progress || *next || || || || 80 || slice18 || 1 || 18 to 20hr || 20 to 75 deg || done || done || done || done || done || in progress || 81 || slice18.2 || || 18 to 20hr || -30 to 20 deg || done || done || *next || || || || 82 || slice20 || || 20 to 22hr || 20 to 75 deg || || || || || || || 83 || slice20 || || 20 to 22hr || -30 to 20 deg || || || || || || || 84 || slice22 || ||22 to 24hr || 20 to 75 deg || || || || || || || 85 || slice22 || ||22 to 24hr || -30 to 20 deg || || || || || || || 86 || slice.top || || 0-24hr || 70 -90 deg || || || || |||| || || || || 75 87 * means this is what is currently planned. this can change prior to the actual load. The ones without a * are the slices already loaded (and their order) 76 88 77 89 The idea is that Heather will load into the dvodb all the exposures (cam stage) with a boresite within the ra/dec range, and all the projection cells within the ra/dec range for a given slice. This means that certainly there will be data outside the edges of the slices within a minidvodb slice. This is ok. 78 90 79 Here is an image of the proposed slice order (with the current skycal progress from 2013-01-15): 80 81 [[Image(ipptopsps.slice.ingest.order.png)]] 82 83 == First slice == 91 Here are 2 images of the proposed slice order (with the current skycal progress from 2013-03-01): 92 93 [[Image(sky.png)]] 94 95 [[Image(add.png)]] 96 97 The goal is to ingest as much of the camera stage as possible, in order to properly calibrate the data 98 ------------------------------------- 99 100 == Heather's Notes on addstar etc == 101 102 what follows now is heather's notes on ingest/slice construction. It will only be interesting to heather (for when other slices are needed), also as a log of what was done 103 === First slice === 84 104 85 105 First slice is slice18, but not entirely (we will only queue from 20 up): … … 115 135 }}} 116 136 117 === Statistics on slice18===137 ==== Statistics on slice18 ==== 118 138 119 139 Here are the statistics on slice18. I wanted to see how ingesting slice18 compares to ingesting the same area of sky with various older surveys (same area of sky) … … 143 163 So what can we take from this? First of all it thinks it took ~7 days to ingest almost all of LAP.slice18 (there are about 40 pending). It feels longer than that, more like 1.5 weeks. Heather suspects that this is due to only having dtime_addstar, and not dtime_total in the addProcessedExp table - so I think it should be longer,but not sure by how much. However, this affect will be throughout all processing (they are all processed the same way). What we can see is that slice18 so far has taking the longest time, but the average time is similar to LAP (V2 and 20110809). So it's a bit slower than ThreePi.V3 (I'm not sure why), but not that much slower... One thing that might cause problems is that ipp005 was both creating the dvo db and serving out files for ipptopsps - does this slow it down? 144 164 145 == slice 18.2==165 === slice 18.2 === 146 166 147 167 This is the bottom half of slice 18. To queue the different parts: … … 167 187 168 188 169 == slice 16.2==189 === slice 16.2 === 170 190 171 191 this is the next slice to be worked on. first step, queue the camera stage, as staticsky is not yet done. … … 177 197 }}} 178 198 179 == slice 8==199 === slice 8 === 180 200 181 201 queue cam stage for slice 8 (1-18-2013) … … 192 212 193 213 194 == slice 8.2 ==214 === slice 8.2 === 195 215 196 216 queue cam stage for slice 8.2 (1-18-2013) … … 209 229 210 230 211 == slice 10==231 === slice 10 === 212 232 213 233 queue cam stage for slice 10 (2-5-2013) … … 229 249 230 250 231 == slice 10.2 ==251 === slice 10.2 === 232 252 cam (2-5-2013) 233 253 {{{ … … 240 260 }}} 241 261 242 == slice 12 and slice 12.2==262 === slice 12 and slice 12.2 === 243 263 244 264 queuing these 2-7-2013: … … 266 286 267 287 268 == slice 14 and slice 14.2==288 ===slice 14 and slice 14.2 === 269 289 270 290 queueing these 2-18-2013 … … 284 304 {{{ 285 305 addtool -pretend -definebyquery -label LAP.ThreePi.20120706 -set_dvodb LAP.20120706 -set_minidvodb_group LAP.slice14.part2.skycal -set_minidvodb -set_label LAP.slice14.part2.skycal -stage skycal -uncensored -ra_min 210 -ra_max 240 -dec_min -35 -dec_max 20 -dbname gpc1 -simple | wc 286 287 306 }}} 288 307 289 308 {{{ 290 309 addtool -pretend -definebyquery -label LAP.ThreePi.20120706 -set_dvodb LAP.20120706 -set_minidvodb_group LAP.slice14.skycal -set_minidvodb -set_label LAP.slice14.skycal -stage skycal -uncensored -ra_min 210 -ra_max 240 -dec_min 20 -dec_max 75 -dbname gpc1 -simple | wc 291 292 }}} 293 294 295 == slice 16 and 16.2 == 310 }}} 311 312 313 === slice 16 and 16.2 === 296 314 297 315 16.2 cam already queued and processed. 16: … … 305 323 {{{ 306 324 addtool -pretend -definebyquery -label LAP.ThreePi.20120706 -set_dvodb LAP.20120706 -set_minidvodb_group LAP.slice16.part2.skycal -set_minidvodb -set_label LAP.slice16.part2.skycal -stage skycal -uncensored -ra_min 240 -ra_max 270 -dec_min -35 -dec_max 20 -dbname gpc1 -simple | wc 307 308 325 }}} 309 326 310 327 {{{ 311 328 addtool -pretend -definebyquery -label LAP.ThreePi.20120706 -set_dvodb LAP.20120706 -set_minidvodb_group LAP.slice16.skycal -set_minidvodb -set_label LAP.slice16.skycal -stage skycal -uncensored -ra_min 240 -ra_max 270 -dec_min 20 -dec_max 75 -dbname gpc1 -simple | wc 312 313 }}} 314 315 316 317 {{{ 318 319 320 }}} 321 322 {{{ 323 324 }}} 325 == slice 18.2 and slice 18 skycal (redo) == 329 }}} 330 331 === slice 18.2 and slice 18 skycal (redo) === 326 332 this was after I first set all the addRuns and minidvodbs related to skycal aside: 327 333 328 [heather@ippc18 ~]$ addtool -definebyquery -label LAP.ThreePi.20120706 -set_dvodb LAP.20120706 -set_minidvodb_group LAP.slice18.skycal -set_minidvodb -set_label LAP.slice18.skycal -stage skycal -uncensored -ra_min 270 -ra_max 300 -dec_min 20 -dec_max 70 -simple -dbname gpc1 | wc 329 0 0 0 330 [heather@ippc18 ~]$ addtool -definebyquery -label LAP.ThreePi.20120706 -set_dvodb LAP.20120706 -set_minidvodb_group LAP.slice18.part2.skycal -set_minidvodb -set_label LAP.slice18.part2.skycal -stage skycal -uncensored -ra_min 270 -ra_max 300 -dec_min -35 -dec_max 20 -simple -dbname gpc1 | wc 331 0 0 0 332 333 334 {{{ 335 addtool -definebyquery -label LAP.ThreePi.20120706 -set_dvodb LAP.20120706 -set_minidvodb_group LAP.slice18.skycal -set_minidvodb -set_label LAP.slice18.skycal -stage skycal -uncensored -ra_min 270 -ra_max 300 -dec_min 20 -dec_max 70 -simple -dbname gpc1 336 }}} 337 {{{ 338 addtool -definebyquery -label LAP.ThreePi.20120706 -set_dvodb LAP.20120706 -set_minidvodb_group LAP.slice18.part2.skycal -set_minidvodb -set_label LAP.slice18.part2.skycal -stage skycal -uncensored -ra_min 270 -ra_max 300 -dec_min -35 -dec_max 20 -simple -dbname gpc1 339 }}} 340 341 342 343 344 345 346 347
