IPP Progress Report for the week 2011-10-31 - 2011-11-06

(Up to IPP Progress Reports)

Eugene Magnier

I have finally finished substantial work on psphotStack to address two outstanding issues. The first concerns the throughput: the radial aperture photometry was taking an unexpectedly long time to run, especially on gpc1 stacks (more so than my tests predicted). After adding some optimizations in the code, I discovered that there was a kind of memory leak acting as the main culprit. The problem was in the glibc level: memory which was being allocated and freed by the IPP functions was not being released to the system by the low-level memory manager. We've seen this kind of problem in certain cases in the past. The solution was to rework the code to avoid a certain sequence of memory allocation. This brought the processing time for one pass of the radial apertures from taking 300-500 seconds to < 1second.

I also addressed the problem whereby the radial moment used to determine the kron magnitudes was being measured in too small of a window for certain classes of galaxies. The algorithm was not going out far enough to encompass the galaxy fluxes. I added code to better identify the boundary where the galaxy surface brightness profile meets the sky level as a starting point, and this gave much better comparisons to sextractors' kron magnitudes.

I spent some time updating pantasks to allow user-specified 'nice' levels for different jobs, and also to allow the server to have a user-defined port range, allowing us to run several pantasks on the same computer (to better manage computing resources).

Serge Chastel

  • Sick Tue+Wed
  • ippdb02 replication is back
  • diff non-unique diff_skyfile_id 162937 investigation
  • ganglia temperature monitoring

Heather Flewelling


Roy Henderson

  • vacation

Mark Huber

  • MD04-r very deep stack release. MD07 exposures finished running through warp, MD07-y refstack and test samples made.
  • continued working through ppStack and ppSub testing
  • Out sick Tuesday, Czar Friday

Bill Sweeney

  • Modified database schema for diff stage to handle case where difftool -definewarpwarp selects two exposures but for differencing but they actually don't have any skycells in common. This was trickier than we expected but in the end we solved the problem.
  • In the process of testing this feature we discovered an inconsistency between the production gpc1 database and the replicated version. An extra row exists in the replicants. Spent several hours with Serge attempting to determine how this might have happened. No conclusions as yet
  • Gathered some M31 warp exposures to demonstrate the severe false streak detection problem that we are having with M31 this year. This involved reprocessing some exposures and writing scripts to copy the data to a directory on one of the stare nodes and run the ppSkycell program to generate binned jpegs.
  • Started working on adding a new feature to the postage stamp server to allow users to select images that have the background correction restored. This has been requested by users looking at large sources. Something that is different here is that we are going to do processing which has not been run before. That is we won't require that a chipBackgroundRun exist for the exposure. This is something new for the postage stamp server, but in this case it appears to be something that we can implement with reasonable effort.

Chris Waters

  • Diskspace: Further work getting diskspace scripts organized to be more descriptive.
  • Sky background: Began comparison of science images and linearity data, to determine why we are not fully correcting the cell-to-cell variations in the background level. Confirmed that stacking the footprint data in two halves yields the same pattern in the final stacks.
  • Storage: organized nebulous commands to add stare nodes into nebulous and target the first copies of stare data to those nodes.
  • LAP: helped unstick LAP processing a few times during the week. Also checked the astrometry failure rate for LAP region H, which is located near the pole. This region has a 6% failure rate, compared with 1-2% for the rest of the sky. This shows that we do have problems solving the astrometry there, but it is not catastrophic.
  • asinh: Updated the header values for asinh scaling to be more descriptive of the actual process to reverse the scaling.