Index: trunk/psphot/doc/footprints.txt =================================================================== --- trunk/psphot/doc/footprints.txt (revision 17710) +++ trunk/psphot/doc/footprints.txt (revision 17710) @@ -0,0 +1,43 @@ + +pmFootprintCullPeaks is very expensive (15 - 30 msec per object) and +must be performed on every peak. We need to improve the speed (even +at the cost of memory) and we need to thread it. + +*** pmFootprints makes use of the mask image, but it does not use the + master mask values. this must be fixed! + +-- here is the algorithm + +** note: this is run once for each footprint... + +* the incoming footprint peak list is already sorted by brightness + +* loop over all peaks + + * choose a threshold which is the peak flux - nsigma_delta*stdev + ? actually, the threshold is using the edge of the containing + subimage? I'm a little confused by this. + + * if the threshold is below a minimum threshold, we drop it + - the code is calling psArrayRemoveIndex for each of these, then decrementing i. + this is inefficient since psArray is not a psList. two possible options here: + + * do not remove the dropped peaks from the array until the full + processing is done, then re-build the array in a single pass. + flag peaks to be dropped (either in the structure or with a + parallel vector) + + * have the code build a separate 'bright peaks' array on the fly + then swap the new peaks out under the pmFootprint. + + * psArrayRemoveIndex is used 3x in pmFootprints.c : review each case + + * the subImage is divided into footprints above threshold about the + current peak. + + - EAM : Need to disect pmFindFootprintsAtPoint. Is it expensive + in this case? Should we be calling the normal pmFootprints + function? + + + Index: trunk/psphot/doc/timing.txt =================================================================== --- trunk/psphot/doc/timing.txt (revision 17710) +++ trunk/psphot/doc/timing.txt (revision 17710) @@ -0,0 +1,84 @@ + + -- threadable? + | + 0.000673 sec NPEAK 0.00 msec / item (505) - ? Find SatStars + 0.231828 sec NPIX 0.00 msec / item (PIX) - ? Replace Background + 0.349591 sec NPIX 0.00 msec / item (2048*4096) - ? Background Model + 0.609246 sec NPIX 0.00 msec / item (2048*4096) - ? Find Peaks + 0.733840 sec NPIX 0.00 msec / item (2048*4096) - ? Find Peaks + 6.076478 sec NPIX 0.00 msec / item (2048*4096) - ? Smooth Images + 7.683749 sec NPIX 0.00 msec / item (2048*4096) - ? Smooth Image + 0.003003 sec NPEAK 0.01 msec / item (505) - ? Source Size + 0.003934 sec NPEAK 0.01 msec / item (??) - ? Source Sizes + 0.019364 sec NPEAK 0.04 msec / item (505) - ? Find Blends + 0.045397 sec NPEAK 0.09 msec / item (??) - ? Replaced Flux + 0.046370 sec NPEAK 0.09 msec / item (505) - ? Add Noise + 0.046680 sec NPEAK 0.09 msec / item (505) - ? Sub Noise + 0.050980 sec NPEAK 0.10 msec / item (505) - ? Replaced Flux + 2.785612 sec NFOOT 0.27 msec / item (10455) - ? Find Footprints + 0.322163 sec NPEAK 0.64 msec / item (505) - ? Rough Class + 0.163793 sec NPEAK 0.66 msec / item (??) - ? Rough Class + 0.820221 sec NFOOT 1.40 msec / item (587) - ? Find Footprints + 2.311492 sec NPEAK 4.58 msec / item (??) - ? Built Models + 2.413531 sec NPEAK 4.78 msec / item (505) - ? Build Models + 1.635674 sec NPEAK 6.54 msec / item (??) - ? Measure Moments + 3.547435 sec NPEAK 7.02 msec / item (505) - ? Measure Moments + 8.118224 sec NSRCS 7.81 msec / item (1040) - ? ApTrend + 4.165990 sec NPEAK 8.25 msec / item (505) - ? Fit PSF (linear) + 4.294968 sec NPEAK 8.50 msec / item (505) - ? Fit PSF (linear) + 4.638916 sec NPEAK 9.19 msec / item (??) - ? Fit PSF (linear) +154.599482 sec NFOOT 14.79 msec / item (10455) - ? Cull Footprints + 1.259951 sec NPSF 17.75 msec / item (71) - ? Fit PSF (non-linear) + 22.009468 sec NFOOT 37.49 msec / item (587) - ? Cull Footprints: + 15.069461 sec NPEAK 38.34 msec / item (393) - ? Fit PSF (non-linear) + EXT (27) + 2.938324 sec NPSF 41.38 msec / item (71) - ? Fit EXT (non-linear) + 2.723019 sec NPSF 53.39 msec / item (51) - ? PSF Residuals + +244.837692 sec complete psphot run: + +Note: + +* PSF Residuals is the most expensive per item, but will always be + performed on a limited number of items (typically < 300 PSF stars). + The maximum impact is thus limited. + +* The non-linear fitting is (not surprisingly) very expensive per + item, but again will only be performed on a limited subset (S/N > + 20?). + +* The Cull Footprints is quite expensive per item, and has the serious + problem that it must be performed on all peaks. This will dominate + all processing time unless we can address it. + +* The linear fitting in this example seems to surprisingly expensive + per item, but I think it is not quite accurate. + +* ApTrend is only measured on a subset of the sources (S/N > XX, max + total number) + +* Measure Moments and Build Models are potentially a substantial drain + since these (probably) should be applied to all objects. Moments is + currently being limited to brighter objects. + +Thread strategies: + + +* tests with wbb4.031117_0154.052.fits test -chip 7 : + +* baseline: + grow : 18.313876 sec + merged : 18.772070 sec + total : 26.958080 sec (dt = 8.18) + +* upgrade to remove psArrayRemove: + grow : 18.327212 sec + merged : 18.778043 sec + total : 25.553486 sec (dt = 6.78; ddt = 1.4) + +* recycle idImage: + grow : 18.383393 sec + merged : 18.848875 sec + total : 24.721451 sec (dt = 5.88; ddt = 2.3) + +(no real difference??) + Index: trunk/psphot/doc/timing.v2.txt =================================================================== --- trunk/psphot/doc/timing.v2.txt (revision 17710) +++ trunk/psphot/doc/timing.v2.txt (revision 17710) @@ -0,0 +1,94 @@ +# msec - multithread? +# item | +15.558560 sec 37.31 NBRT 417 : y non-linear fits + 0.000736 sec 0.00 NPEAK 550 : ? satstar blends + 0.005947 sec 0.00 NPEAK 1315 : ? source sizes + 0.076172 sec 0.01 NPEAK 7636 : ? set footprint ids (NPIX?) + 0.003383 sec 0.01 NPEAK 550 : ? source sizes + 0.019354 sec 0.04 NPEAK 550 : ? other blends + 0.050572 sec 0.08 NPEAK 627 : ? set footprint ids + 0.049778 sec 0.09 NPEAK 550 : ? replaced models + 0.051736 sec 0.09 NPEAK 550 : ? add noise + 0.051929 sec 0.09 NPEAK 550 : ? sub noise + 0.055094 sec 0.10 NPEAK 550 : ? replaced models + 0.152204 sec 0.20 NPEAK 765 : ? rough class + 0.8 sec 0.26 NPEAK 3100 : ? merge footprints + 0.268983 sec 0.49 NPEAK 550 : ? rough class + 0.7 sec 1.25 NPEAK 560 : ? cull peaks + 2.092846 sec 2.74 NPEAK 765 : ? moments + 8.9 sec 3.05 NPEAK 2914 : ? cull peaks + 3.339909 sec 4.37 NPEAK 765 : ? built models + 2.572153 sec 4.68 NPEAK 550 : ? build models + 3.816792 sec 6.94 NPEAK 550 : ? moments + 9.337548 sec 7.10 NPEAK 1315 : ? measure magnitudes : for 1315 objects (643 with apertures) + 4.301656 sec 7.82 NPEAK 550 : ? linear fit + 4.429642 sec 8.05 NPEAK 550 : ? linear fit + 4.947403 sec 9.00 NPEAK 550 : ? linear fit + 0.2 sec 0.02 NPIX 8300 k : ? found grown footprints: + 0.205680 sec 0.02 NPIX 8300 k : ? find footprints + 0.211664 sec 0.03 NPIX 8300 k : ? replace background + 0.348357 sec 0.04 NPIX 8300 k : ? subtracted background + 0.598805 sec 0.07 NPIX 8300 k : ? find peaks + 0.599952 sec 0.07 NPIX 8300 k : ? find peaks + 0.723508 sec 0.09 NPIX 8300 k : ? found footprints: + 1.4 sec 0.17 NPIX 8300 k : ? found grown footprints: + 0.809106 sec 12.64 NPSF 64 : ? psf fit + 2.155031 sec 34.21 NPSF 63 : ? full fit + 2.592620 sec 74.07 NPSF 35 : ? psf residuals + 5.931090 sec 0.71 sNPIX 8300 k : ? smoothed signficance + 7.780419 sec 0.94 sNPIX 8300 k : ? smoothed signficance +12.947970 sec 1.56 sNPIX 8300 k : ? convolved with grow disc: +14.525486 sec 1.75 sNPIX 8300 k : ? convolved with grow disc: + +Notes: + +Note: + +* PSF Residuals is the most expensive per item, but will always be + performed on a limited number of items (typically < 300 PSF stars). + The maximum impact is thus limited. + +* The non-linear fitting is (not surprisingly) very expensive per + item, but again will only be performed on a limited subset (S/N > + 20?). + +* the linear fits are the next most costly, and will dominate the + timing since all sources must be so fit, and multiple times. + +* measure magnitudes is expensive, but only runs once + +* measure moments is expensive, but only runs once (and/or on a bright subset?) + +Adding it up: + +GPC1 : 10^11 det for 60*5000* 60 chips --> 5500 det / chip + : 23 megapix + +* cost per object: + + 56.5 msec -> 310 sec for detection analysis (on kawelu, not optimized) + +* cost for bright detections + + ~ 35 sec (for 1000 bright sources) + +* cost per megapixel: + + 5.5 sec + + -> 126 sec for GPC1 + +* other overhead (psf construction, background, smoothing) + + -> 20 sec + +** total : 490 sec (kawelu, not optimized, 1.0GHz!) + +** ~ 220 sec optimized +** ~ 100 sec on 2.3GHz machines +(probably there are other gains on the real hardware: memory bandwidth, ?) + +assuming 100sec is all we get, we need ~1.5 cores per chip to keep up at 60sec / exposure (average) + +120 cores for ppImage (chip) + Index: trunk/psphot/doc/timing.v3.txt =================================================================== --- trunk/psphot/doc/timing.v3.txt (revision 17710) +++ trunk/psphot/doc/timing.v3.txt (revision 17710) @@ -0,0 +1,39 @@ +0.000408 sec found 0 satstar blend peaks, leaving 551 sources: +0.002058 sec measure source sizes for 551 sources: +0.003553 sec measure source sizes for 1302 sources: +0.009676 sec identified 36 blended objects: +0.026266 sec replaced models for 551 objects: +0.026892 sec sub noise for 551 objects: +0.026924 sec add noise for 551 objects: +0.029050 sec replaced models for 551 objects: +0.032975 sec set footprint array IDs: +0.047061 sec set footprint array IDs: +0.080533 sec rough classification: +0.107024 sec replace background flux : +0.111200 sec found 631 footprints: +0.132417 sec rough classification: +0.182467 sec subtracted background model: +0.2 sec cull peaks +0.297289 sec 2676 peaks: +0.3 sec rest of foot +0.300492 sec 849 peaks: +0.364908 sec found 7613 footprints: +0.393265 sec measure ensemble of PSFs: +0.405241 sec build median image: +0.454711 sec measure ensemble of PSFs: +0.680027 sec measure ensemble of PSFs: +0.900696 sec 751 sources, 290 moments, 9 failed: +1.1 sec rest of footpr +1.147541 sec built models for 551 objects: +1.289976 sec generate residuals for 35 objects: +1.456877 sec built models for 751 objects: +1.5 sec select psf model (ex resid) +1.661166 sec 551 sources, 551 moments, 8 failed: +2.988601 sec built smoothed signficance image: +3.677652 sec built smoothed signficance image: +4.354904 sec measure magnitudes : for 1302 objects (634 with apertures) +5.8 sec cull peaks +6.594287 sec convolved with grow disc: +7.413455 sec convolved with grow disc: +7.616990 sec fit models: for 417 objects (354 psf, 31 ext, 32 failed, 134 skipped) +55.393567 sec complete psphot run: