IPP Progress Report for the week 2099.11.16 - 2209.11.20

(Up to IPP Progress Reports)

Eugene Magnier

We have been focusing this week on three main issues of concern by the KPAT/DRAVG : photometry in stack, astrometry issues, and crosstalk. Paul has made good progress at understanding the behavior of the photometric quality of the stacks, and at this point, I believe that he and Nigel agree that the pixel values are OK: they have both shown that aperture photometry comparisons between warps and stacks have scatter which is small (where not photon limited) and linear. For stacks with good IQ input images, Paul showed that the psf photometry is substantially more stable than those with varying input IQ, though there is still more scatter in the psf photometry than expected from signal-to-noise trends. This last point highlights that the scatter is probably being driven by the mismatch between PSF and model coupled to the correlated errors.

Chris has finished the crosstalk masking, and it is incorporated in the standard analysis.

I have been working on the astrometry issues, both large-scale and small-scale failures. for the large-scale failures, I have made several fixes that substantially reduce catastrophic failures and give more robust solutions to avoid the small-scale errors (see below). The code for testing and retrying or rejecting badly fit chips based on the rest of the chips is working, but is not currently being used for GPC1: currently, the guess is coming from a linear model of the focal plane, which is not really accurate enough for this rejection. I need to generate a full-scale model, but I have not fully tested that the full non-linear model behaves correctly in all cases. I am going to put that on a lower priority until we have more statistics on the current catastrophic failure rates. One remaining fix that is needed in this regard is to ensure that warp is not including chips which have failed: there have been reports of known bad astrometry solutions being used in the warping analysis.

There were a number of poor astrometry results showing up in difference images pairs for run 3 data, especially in the low Galactic latitude 3pi fields. After working on the iterative clipping process to be robust against outliers in these cases, I finally realized that these problems were in fact caused by a bug in PSPHOT -- these images where processed before a number of changes over the summer to the psphot source size analysis, and many stars in these images were identified as cosmic rays. The code then masked the cores of these stars, and re-fitted sources in the wings of the stars. As a result, there were large numbers of consistently offset sources at 2.5 pixels from the centers of real reference stars in one of the 4 cardinal X,Y directions. running psastro on a newer psphot analysis of the same images gave much better results.

We also had a hardware incident over the weekend: the database machine ippdb02 crashed complaining about a hardware error. The machine did not reboot on power-cycle, and we were concerned that the failure may have been serious. Cindy & Bill were able to get the machine back up after re-setting the BIOS values, and it seems the database is intact. However, this gave us a good opportunity to set up an alternate Nebulous installation on ippdb00, something which has not been done for our production machines since Josh left.

  • psastro fixes:
    • improve the test for the allowed order for a given number of stars, and make the requirements tighter.
    • iteratively remove clumps of objects in the raw and ref tables used for the astrometric match and analysis. This avoids catastrophic failures due to nebulosity and galaxies.
    • add a feature to require a single (closest) reference match in the fitting after the first N clipping iterations. This prevents solutions driven by outliers in crowded fields.
    • some minor tuning of the clipping radii for GPC1

Heather Flewelling

  • chased down lots of processing problems
    • thousand of diffs failed, a full ipp016 was the cause, and some incorrect queries also caused problems. Bill and Paul helped out to debug this mess.
    • a few failures from each stage, mostly caused by the 0 byte neb bug. These have been dealt with, and Chris put in a fix that prevents them. The 0 byte bug often occurs when there is a high load on a machine.
  • manned summitcopy and registration again, since there was stuff to download. investigated registration failures.
  • figured out which raw files exist only on cabinet 2.
  • started copying megacam files into nebulous, about halfway done. (do 'neb-ls megacam' to find them)
  • started md09 processing. working on diffs and greater. things are distributing. yay.
  • started md10 processing. chips are in, should go through to warp this weekend.
  • created a nebulous volume log on wiki. http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/NebulousVolumes
  • mask fractions for random chips and warps in throughput.20091104:

Bill Giebink

  • Got DIMM operational by repairing relays; serviced DIMM
  • Located and prepared requisitions for replacement and spare relays.
  • Attended scaffold training.
  • Spent a small amount of time on realtest.
  • Worked with Tom and Steph to setup a laptop to serve accelerometer data.

Paul Price

  • Stack photometry
    • Warp compared with warp: 0.016 mag (aperture) to 0.024 mag (PSF) scatter
    • Warp compared with stack: 0.076 mag (PSF) scatter
    • These STS warps have bad PSFs! Generating own stack with "good" inputs failed because bad PSFs despite iq_m[234] being decent: double images and/or triangular PSFs.
    • Kepler is sparse in terms of PS1 coverage; ditto SA92.
    • Using Gene's flattest exposures. Photometry comparison residuals:
      • warp-warp PSF 0.010
      • warp-warp Aper 0.010
      • warp-stack PSF 0.016
      • warp-stack Aper 0.014
    • Very pleased to see Nigel's values photometry comparison plots: good aperture photometry suggests stack pixels are fine.
    • Using simtest to verify that stack pixels are fine.
    • Discovered RA=0 boundary bug in DVO/getstar
    • 1 simtest stack failed; no time to investigate
    • Simtest run in /data/ipp022.0/price/stack/simtest
  • Fixed headers (WCS, concept values, software versions) for unconvolved stacks.
  • Threading for pmReadoutFake, trunk@26202. If there are problems with this while I'm away, "svn merge -c -26202" should remove this change.
  • Fixed division by zero in magic_tree.pl that was preventing STS processing.
  • Preparing for UK trip.

Bill Sweeney

  • Worked with people at MPIA to get the current version of the IPP built and installed at their site.
  • Installed a data store at MPIA for them to use to submit postage stamp requests
  • Modified the ipp data store tools to work through a proxy server.
  • Helped monitor and debug issues in the destreaking and distribution portions of the pipeline.
  • Figured out why diffs were taking so long to bundle. There was a bug in the pending query that caused runs to get bundled multiple times. Fixed this and deleted the duplicates from the data store.
  • Finished postage stamp server "get_image" functionality. This will allow users to randomly select and download gpc1 data through the postage stamp server.
  • Added the ability to select psf, cmf, and background model image to be included in postage stamp data products.
  • At this point the processing portions of the postage stamp server are functionally complete. (The documentation needs to be updated though).

Chris Waters

  • Crosstalk: Finished and tested crosstalk masking in psastro. On test exposure, correctly identified and masked all crosstalk artifacts known to follow the rules defined by John Tonry.
  • Data quality statistics: Wrote new task/script/tool group to create fits table bundles of exposure statistics for transfer to OTIS. All code is written, but no testing has been performed to confirm functionality.