IPP Progress Report for the week 2011.02.28 - 2011.03.04

(Up to IPP Progress Reports)

Eugene Magnier

I ran psphotStack on the 8-way MD04 stacks from last week. It took a couple of iterations to get reasonable answers. In the first round, the memory footprint exploded and crashed several of the machines used for the analysis. I discovered that the 'chisq' detection image (combination of all input filter images) was not correctly defined. I temporarily disabled this image as part of the detection process. This also uncovered an interesting memory waste : when psphotStack generates source entries for the undetected sources in the other filter images, the footprint analysis always uses the largest footprint as the container for all new footprints. This is excessive and can lead to large memory usage in some cases. I have not yet made a final fix on the latter problem, but by avoiding the chisq image for now we can work past it to continue testing.

I addressed a number of other issues in the outputs from psphot, psphotStack, and ppStack. Specifically: I fixed the the calculated chisq values for all objects -- there were a few inconsistencies, but the worst was that the unweighted non-linear fits were making an ill-advised correction to the chisq to account for not using the weights in the fitting. I changed this to simply recalculate the chisq with the correct weights after the fit was completed. The chisq distribution for all sources now seems quite sensible (centered around 1.0 for most objects, though growing for the brighter stars where the systematic differences between the model and the data become important).

I also fixed the output headers of the stacks to correctly set CELL.SATURATION, taking into account the image flux re-scaling. I also wrote up an analysis of the steps in ppStack which affect the flux scaling and zero points: Stack. I also added more information to the output stack headers: the names of all input images, their exposure times, airmasses, reported zero points and any flux scaling applied to them before combination.

I looked into cases where the ppStack target PSF is oddly high. This appears to happen because the PSF model from one or another of the input warps is poor. Looking at this more closely, I believe that the pswarp psf model is being generated from sources whose positions are held fixed; any small error in the astrometry then couple badly into the resulting PSF model.

Finally, I examined Celestial North Pole images which were failing in psphot in an interesting way: they were growing to 20GB and finding 10s of thousands of sources. The cause for this is clear: these images have very poor seeing and interesting variations in the background from clouds. The background model is not following the actual background structures perfectly, and leaving behind some large, low-surface brightness features in the background-subtracted image. When this image is smoothed with the large PSF for detection, these low-surface brightness regions show up as significant extended flux areas, and many sources are detected. Michael Wood-Vasey has suggested some additional filtering that could be used to check on the significance (as PSFs) of the sources found in these areas.

Serge Chastel

  • MOPS tool for processing: specs + development
  • MySQL databases optimization. Full report here. Conclusions:
    • Nebulous optimization took 2 days.
    • Nebulous huge tables sizes have decreased of 20% / 25%.
    • GPC1 optimization took about 2 hours
    • GPC1 tables have globally shrunk in size by 27%.
    • It would be highly interesting to optimize in one week?/month? to see if optimization lasts so long.
  • Other:
    • apache log of ippdb00 has been moved to /export/ippdb00.0
    • ippdb01 MySQL should be cleaned (1648 tables in total while gpc1 is 96 tables).
    • ippdb02 doesn't work as expected for backup.

Heather Flewelling

  • conference - telescopes from afar
  • found one of MD07.jtrp stacks failed with: fix this code: z1 should not be nan for 0.401352 , located here: gpc1/MD07.jtrp/MD07.V2/skycell.086/MD07.V2.skycell.086.stk.238397.log. This still needs to be fixed...
  • answered questions for Ken Smith (photfest)
  • investigated new date based ippconfig - static masks are properly picked up, date based pattern.cell/row are not. This was partly due to them not being installed (Makefile needed some help), but once installed they still don't work quite right.
  • tried sneaking in MD08.jtrp into processing, it was removed.
  • created a script to find the nearest raw exposures for a given ra/dec. Need to check this in.

Roy Henderson

  • PSPS
    • added user instructions to ippToPsps documentation (as I realized, during my recent absence, that no one but me could run it)
    • some time pulling various images, stacks and cmfs for Jim for MD04. Time figuring out filter type of new stack cmfs that were missing headers
    • 2 new passes of stacks from Gene. 1st:
      • extensions names changed from demo file
      • all fluxes invalid in psf extension
      • SkyChip?.xsrc and SkyChip?.xrad extensions empty
    • 2nd:
      • now populating ApFluxStDev? from SkyChip?.xrad
      • headers empty
      • work on xfit extension: Sersic, deV and Exp models. Plan to load FITS straight into a Db in order that I can cleanly query the data. Alternative is to write lots of code to do what databases already do.
    • walk-through of nice new DXLayer from Conrad. Loading some test batches to test it works. It does!
  • IPP
    • added 'interested servers' list to XML config for roboczar
    • added comments to Config class and XML config
  • Other
    • sick on Monday
    • general catch-up after two weeks off
    • lots of mailing-list admin

Bill Sweeney

  • Spent a day learning to use DVO and analyzing why the PanPlanets? project is reporting bad astrometry. We belive that the problem is due to stars getting damaged by burntool correction. The large number of brigth stars in the STS fields makes these exposures more sensitive to this problem. Gene has an idea on how to improve the situation.
  • Gathered up detrend images used for STS processing and put them on the data store for MPG to download.
  • Fixed problem in dvoImagesAtCoords where it found that a point on the sky touched two chips.
  • Wrote a script to parse the host configuration files to report how hosts are being targeted for procesing
  • Fixed some problems with destreak cleanup for non-replace mode runs (raw stage).
  • Found a bug in psphot that caused some older data to fail updates. Implemented fix suggested by Gene.
  • Analyzed several bursts of postage stamp errors. Most are due to user error. There are some cases where transient "file not available" errors are treated as "file gone errors".

Chris Waters

  • Out two days due to illness.
  • Diffraction spikes: identified that the stars with the largest error in mask positions also have the largest proper motion. Since the actual star image position isn't where the masking code thought it would be, the masks miss. There is still a small 0.25 degree offset that will need to be resolved.
  • Image noise properties: I checked the distribution of un-masked pixel values for a dark exposure, in an effort to characterize the level to which the image noise is non-Gaussian. The best fit noise level is consistent with that calculated from RDNOISE / GAIN, although the distribution larger-than-Gaussian beyond 5 \sigma.
  • Detectability server: Re-read and reminded myself of how the detectability server works to allow for bugfixes and improvements to be made next week. This should implement SSdiff queries and provide the original psphotForced CMF results in the response.