| | 7 | |
| | 8 | We (Chris, Heather, and I) got back from Belfast and spent part of the week getting caught up on issues here. One major issue we are trying to deal with: we have about 100TB of storage used up by difference image output products that we thought were being deleted. We are trying to delete these to clear up needed space, but it has been slow (see Chris' report). Relatedly, we have been seeing a lot of sluggishness in the processing lately, and spent some time trying to understand this. I am a bit concerned that the Nebulous mysql is getting bogged down by all of the interaction due to this cleanup and other data juggling needed to balance the storage usage on the cluster. Some of the sluggishness seems to have also been caused by a hung NFS server. Over the weekend, I shutdown and restarted all of the processing as well as the Nebulous mysql, and this seems to have helped, returning the processing to normal speeds. However, I still suspect that the Nebulous mysql could see some efficiency improvements, perhaps if we partition the very large table of disk instances. We will attempt some experiments on this (offline) soon. |
| | 9 | |
| | 10 | The overall goal now is to address all of the known concerns with the processing in order to be able to start The Grand Reprocessing. In Belfast, a core group of difference-image-interested people (Armin Rest, Michael Wood-Vasey, Mark Huber, Ken Smith, Maria Teresa Botticella and I) agreed to generate a consistent test set of difference images and detections using both the CfA 'photpipe' code and the IPP 'ppSub' and have a focused meeting at Harvard comparing the results. Clearly, it would be best to address the issues already identified at Belfast before generating a large set of results. Thus, in the short term, my main development focus is to make some tweaks to the difference imaging algorithms based on issues raised to date. Ken Smith pointed out that there are still times when the dual convolution kernels do not match as well as one might imagine. I spent part of last week checking for possible causes, and identified a couple of low-level issues. First, the dynamic range of the matrix equation used in the psf matching code was very large, and seems to have been susceptible to floating point round-off errors. I adjusted the relative scaling of the kernel terms to reduce the dynamic range and this improves things. Second, the penalty function to prevent the dual convolution from expanding the output PSF seems to be too sensitive. I am reviewing that part of the code to see how this can be made more reliable. Maria Teresa Botticella also pointed out some cases where the difference image statistics intended to identify dipoles and related garbage do not seem to behave quite as expected. I'll be looking into this after the penalties. |
| | 11 | |
| | 12 | Another area of development to be addressed before reprocessing is to incorporate feedback on the extended source analysis. Nigel Metcalf, John Lucey, and Peter Draper have started to examine the extended source parameters generated by the IPP, and when they have more complete results, we will be able to fold those suggestions back into the code. The final area of core data analysis needed for reprocessing, for which we are getting feedback from the consortium, is in the star/galaxy characterization. Eddie Schlafly has started to provide comparisons with SDSS using the recent (2010.08.23) update to the IPP classifications. Hopefully, he will be able to help me to adjust the conversions from measured moment and magnitude offsets (relative to the PSF model) into star/galaxy probabilities. |
| | 13 | |
| | 14 | There are two other issues to be dealt with before reprocessing: hardware upgrades and detrend upgrades. We received the proposals for the new computers (15 compute nodes), and will be cutting a PO very soon. We are also moving forward on purchase of more 2TB disks to upgrade 4 more wave 1 nodes, to yield an additional 120T net. In the area of detrend, Chris & Heather are both working on their related areas: Chris is working on the non-linearity correction code and Heather is testing the modification to the code needed to make a better dark model. These should be finished in the coming week, at which point we need to update our masks. |
