| | 1 | == IPP Progress Report for the week 2009.10.26 - 2009.10.30 == |
| | 2 | [[PageOutline]] |
| | 3 | |
| | 4 | (Up to [wiki:IPP_Progress_Reports IPP Progress Reports]) |
| | 5 | |
| | 6 | === Eugene Magnier === |
| | 7 | |
| | 8 | === Heather Flewelling === |
| | 9 | |
| | 10 | * processing several labels: |
| | 11 | * Cal.2009106 |
| | 12 | * SVS.Run4.r.20091027 (chip and cam only) |
| | 13 | * ThreePi.Run3.r.20091027 (chip and cam only) |
| | 14 | * STS.20091027 |
| | 15 | * MD08.y.20091029 |
| | 16 | * and a few other labels for diffs and other stages |
| | 17 | |
| | 18 | * destreaking md08 label |
| | 19 | |
| | 20 | * published sweetspot label |
| | 21 | |
| | 22 | * investigating failures on stdscience |
| | 23 | * nfs problems/glitches |
| | 24 | * log files and other files with 0 bytes and funny write protected state blocking nebulous/pantasks from proceeding (from past nfs issues) |
| | 25 | * programming errors: I'm finding them mostly on pswarp, 2 different types of pswarp errors |
| | 26 | * diffs on sweetspot: some don't have faults? |
| | 27 | * some burntool failures (on the calibration images, they were not burntooled for various reasons, this was fixed) |
| | 28 | * no detrends found for some of the calibration images: those were taken in w.00000 band, so I just dropped those images. |
| | 29 | * chips on MD08: some don't have faults? |
| | 30 | |
| | 31 | * checked that all important (ie, raw images) that we care about have been replicated off of ipp008 |
| | 32 | |
| | 33 | * helped natalia with some simtest questions |
| | 34 | |
| | 35 | === Bill Giebink === |
| | 36 | * Continued on "realtest" |
| | 37 | * Added card/disks to ippc018/c019; swapped memory on ipp005 |
| | 38 | |
| | 39 | === Paul Price === |
| | 40 | |
| | 41 | * Stacking (branches/pap) |
| | 42 | * Investigating rejection behaviour: some stars rejected from all |
| | 43 | images, some features on a single image (e.g., burn) can result in |
| | 44 | pixels from all images being rejected, scattering of pixels rejected on |
| | 45 | a single image |
| | 46 | * Add variance based on PSF-matching chi^2^ when stacking: this feature |
| | 47 | was recently turned off when I thought I had the convolutions behaving, |
| | 48 | but it seems it's necessary for getting the variance right (chi^2^ ~ 4). |
| | 49 | With this feature on again, the rejection in the cores of stars (from |
| | 50 | which the parameters are measured!) is much suppressed. |
| | 51 | * Now a faint satellite streak gets through! Am I |
| | 52 | double-renormalising the variance? |
| | 53 | * Streak can still be seen over a much shorter length (e.g., only one |
| | 54 | other input image) before I made this change. |
| | 55 | * Also more other faint artifacts (e.g., burns) |
| | 56 | * I was double-renormalising the variance; fixed. Now tweaking |
| | 57 | parameters (COMBINE.REJ = 2.0, COMBINE.SYS = 0.1) |
| | 58 | * Realised that "safe" mode is not doing what we want. After |
| | 59 | rejection, pixels come in three categories: tested and good, tested and |
| | 60 | rejected, and not tested. The code currently does not recognise the |
| | 61 | third, which is a distinct state because we don't want these pixels |
| | 62 | grown, as we do for rejected pixels. This cannot be fixed merely by |
| | 63 | using the "safe" combination because that would discard the "tested and |
| | 64 | good" pixels that have only a single unrejected input but are good |
| | 65 | because they have survived the testing process. Need to add a new state |
| | 66 | into the combination process. Now I add these pixels straight into the |
| | 67 | "reject" list. |
| | 68 | * Rejection of stars is much reduced now. Remainder seems to be due |
| | 69 | to images having slight shifts. |
| | 70 | * We should be able to take shifts out through PSF-matching. However, |
| | 71 | there are shifts in the fake images as well. Can attempt to solve for |
| | 72 | dx,dy in the course of ZP measurement. |
| | 73 | * Measured image shifts are not large (0.3 pixels max) --> must be |
| | 74 | incoherent shifts, which we can't do anything about in the stacking |
| | 75 | process. |
| | 76 | * Attempting to get a handle on the variance behaviour: renormalising |
| | 77 | from the background gives us a different factor (~1.3) than the |
| | 78 | PSF-matching chi^2^ does (~4). How should we normalise the variance? |
| | 79 | * Playing with softening the errors in the PSF-matching. Perhaps the |
| | 80 | PSF-matching can give us a softening parameter for the stacking? |
| | 81 | * Softening parameter wasn't helping at all (I expect most of the |
| | 82 | signal in the chi^2^ was coming from the background). It seems that the |
| | 83 | PSF-matching had a lot of outliers that need to be removed with more |
| | 84 | iterations. Then chi^2^ ~ 2.6 which is a bit closer to the value from |
| | 85 | the background (1.3). I think the difference comes from the covariance |
| | 86 | matrix (~0.5). |
| | 87 | * But this still means we're double dipping: tweaking the variance |
| | 88 | twice (even if it's approximately the same factor each time). |
| | 89 | * Determined with Gene that we need to soften the variance --> |
| | 90 | variance + frac * flux^2^ (equivalent to imposing a minimum fractional |
| | 91 | error). Actually, was already doing that, though it was set a little |
| | 92 | low (0.03). |
| | 93 | * Sources near masked areas are prone to trouble. Let's throw out the |
| | 94 | CONV.POOR data from the stack. Aha! Now the masked stars seem to be |
| | 95 | due to lack of overlaps, which we can't control. Perhaps we can throw |
| | 96 | out CONV.POOR only in the case that something is inconsistent. |
| | 97 | * Updated combination algorithm to throw out all suspect pixels as the |
| | 98 | first step in rejection (only done if something is inconsistent). |
| | 99 | * Added combination case to do better testing for 3 input pixels (lots |
| | 100 | of pixels when there's only 4 inputs; and it tends to go a little insane |
| | 101 | with the general case, throwing out all inputs) |
| | 102 | * Reworking mammoth function combinePixels() into multiple smaller |
| | 103 | functions |
| | 104 | |
| | 105 | === Bill Sweeney === |
| | 106 | |
| | 107 | * fixed problem in distribution related to warp versus warp diffs |
| | 108 | * wrote a new program to undo background subtraction in chip processed images. (This was |
| | 109 | requested by the M31 project). |
| | 110 | * Started some work optimizing the distribution processing. Found and reported an issue with the distribution |
| | 111 | of skycell data products on the cluster. (Since fixed by Gene.) |
| | 112 | * Debugged some problems with users' postage stamp requests. (I still need to provide better error reporting). |
| | 113 | * Built a simple web page and associated script to show the postage stamp server status. |
| | 114 | * worked with Gene and Heather to build and install a special tessellation for the STS survey. |
| | 115 | * fixed a bug in the distribution client related to database dump file format change made just before |
| | 116 | vacation. |
| | 117 | |
| | 118 | === Chris Waters === |
| | 119 | |
| | 120 | * Fringe. Noted that fringes aren't being fully removed, with the |
| | 121 | residual correlated somewhat with time of night. The scales match |
| | 122 | well for data taken around the time the master fringe data was |
| | 123 | taken, suggesting the deviation is due to changes in signal from |
| | 124 | the sky. |
| | 125 | |
| | 126 | * Stack reviewing. Got up to speed on the current issues with the |
| | 127 | stacking. Noted that burntool masks weren't being respected by |
| | 128 | stacking (now fixed by Paul Price). Looked at the photometric |
| | 129 | consistancy between stacks and warps, determining that the phphot |
| | 130 | errors aren't scaling as expected between these steps. Began work |
| | 131 | on a simtest run to check this with data that doesn't have to worry |
| | 132 | about telescope reality. |
| | 133 | |
| | 134 | * Crosstalk. Looked at a large sample of potential crosstalk ghosts, |
| | 135 | and found that roughly 80% match John Tonry's predictions. |
