IPP Progress Report for the week 2010.09.06 - 2010.09.10

(Up to IPP Progress Reports)

Eugene Magnier

We (Chris, Heather, and I) got back from Belfast and spent part of the week getting caught up on issues here. One major issue we are trying to deal with: we have about 100TB of storage used up by difference image output products that we thought were being deleted. We are trying to delete these to clear up needed space, but it has been slow (see Chris' report). Relatedly, we have been seeing a lot of sluggishness in the processing lately, and spent some time trying to understand this. I am a bit concerned that the Nebulous mysql is getting bogged down by all of the interaction due to this cleanup and other data juggling needed to balance the storage usage on the cluster. Some of the sluggishness seems to have also been caused by a hung NFS server. Over the weekend, I shutdown and restarted all of the processing as well as the Nebulous mysql, and this seems to have helped, returning the processing to normal speeds. However, I still suspect that the Nebulous mysql could see some efficiency improvements, perhaps if we partition the very large table of disk instances. We will attempt some experiments on this (offline) soon.

The overall goal now is to address all of the known concerns with the processing in order to be able to start The Grand Reprocessing. In Belfast, a core group of difference-image-interested people (Armin Rest, Michael Wood-Vasey, Mark Huber, Ken Smith, Maria Teresa Botticella and I) agreed to generate a consistent test set of difference images and detections using both the CfA 'photpipe' code and the IPP 'ppSub' and have a focused meeting at Harvard comparing the results. Clearly, it would be best to address the issues already identified at Belfast before generating a large set of results. Thus, in the short term, my main development focus is to make some tweaks to the difference imaging algorithms based on issues raised to date. Ken Smith pointed out that there are still times when the dual convolution kernels do not match as well as one might imagine. I spent part of last week checking for possible causes, and identified a couple of low-level issues. First, the dynamic range of the matrix equation used in the psf matching code was very large, and seems to have been susceptible to floating point round-off errors. I adjusted the relative scaling of the kernel terms to reduce the dynamic range and this improves things. Second, the penalty function to prevent the dual convolution from expanding the output PSF seems to be too sensitive. I am reviewing that part of the code to see how this can be made more reliable. Maria Teresa Botticella also pointed out some cases where the difference image statistics intended to identify dipoles and related garbage do not seem to behave quite as expected. I'll be looking into this after the penalties.

Another area of development to be addressed before reprocessing is to incorporate feedback on the extended source analysis. Nigel Metcalf, John Lucey, and Peter Draper have started to examine the extended source parameters generated by the IPP, and when they have more complete results, we will be able to fold those suggestions back into the code. The final area of core data analysis needed for reprocessing, for which we are getting feedback from the consortium, is in the star/galaxy characterization. Eddie Schlafly has started to provide comparisons with SDSS using the recent (2010.08.23) update to the IPP classifications. Hopefully, he will be able to help me to adjust the conversions from measured moment and magnitude offsets (relative to the PSF model) into star/galaxy probabilities.

There are two other issues to be dealt with before reprocessing: hardware upgrades and detrend upgrades. We received the proposals for the new computers (15 compute nodes), and will be cutting a PO very soon. We are also moving forward on purchase of more 2TB disks to upgrade 4 more wave 1 nodes, to yield an additional 120T net. In the area of detrend, Chris & Heather are both working on their related areas: Chris is working on the non-linearity correction code and Heather is testing the modification to the code needed to make a better dark model. These should be finished in the coming week, at which point we need to update our masks.

Serge Chastel

  • Off on Tuesday
  • IPP: not a lot this week (I worked on PSPS). I modified the script that displays active processes running on the cluster. I'll build up stats on Monday.
  • PSPS: checking the contents of FrameMeta? and ImageMeta? works with one of Roy's data set (29152). I'm working on Detection. What to do next? SkinnyObject? and ObjectCalColor? in nominal case. I will then have to build up faulty samples to check if the tests are ok when something bad happens. I will also try to build up _tiny_ representative samples to speed up the tests. Roy's large original samples could be used in the future for performance tests (if it is required).

Heather Flewelling

  • returned from UK late monday night
  • czar tuesday and friday
    • machines kept going down
    • registration/summit copy has been slow for many days
    • dravg/mops has had trouble downloading files (in part due to a couple of troublesome machines going down)
  • ifaps1 liason duties - answering questions, helping out users (tonry, ebeling, dixon, deacon)
  • ppMerge tests - problematic, still working on this.

Roy Henderson

  • Vacation all week.

Bill Sweeney

  • Examined the ThreePi?.V1 dvo database. discovered that zero points were still not being applied. Also found that several exposures were added multiple times.
  • fixed bugs in addstar application of zero points. Tested code.
  • wrote a script to extract zero points from the gpc1 database and create a setphot input file.
  • started setphot run on the ThreePi?.V1 dvo database. This was interrupted by ipp004 disk problems.
  • wrote script to parallellize rsync of ginormous dvo databases for backup.
  • finished and integrated tools to simplify running of warp_skycell.pl and diff_skycell.pl from the command line for use in remaking corrupted files or running without threads
  • debugged and fixed a problem with difftool -listrun that was breaking lookups done for detectability queries.
  • Upgraded the postage stamp sever upload page to accept detectability queries.
  • created pstamp.revert.dependent to reduce the amount of operator intervention needed for the postage stamp server.
  • fixed problem with magicdstool -advancerun which caused failures to update the database to cascade.
  • rebuilt 335 warpRuns so that the M31 warp variance images could be posted.(I accidentally cleaned them up before Johannes finished copying the files). Built the data store file sets by copying the files this time so that they will be unaffected by cleanup.

Chris Waters

  • Travel back from Belfast and a day to recover.
  • Detectability Server: bug fixes to prevent request rows that match no images from corrupting the log files. Fixed parser to handle comments that caused previous version to fail. Added ability to request stack-stack and warp-stack diffs from the detectability server. Discovered set of diff-related bugs in the postage stamp/ipptools code that prevents diff detectability requests from succeeding.
  • Diskspace: worked on cleaning old inverse diff destreak images that are currently taking up ~100TB of disk space. Parallelized the process as much as possible without overloading the nebulous server.
  • Nebulous: investigated why the nebulous server was so heavily loaded and slow. No clear answer.