IPP Progress Report for the week 2010.10.04 - 2010.10.08

(Up to IPP Progress Reports)

Eugene Magnier

I spent much of last week working on incorporating lessons from photfest into my branch of the IPP code, and attempting to address some related issue encountered along the way. I updated the measurement of the 'dipole parameters' (DIFF_FRATIO, etc) to NOT clip pixels with abs(S/N) < 3 (we mis-understood Armin's definition). Eddie Schlafly and I iterated further on the cosmic ray and galaxy separation issue. For the cosmic ray detection, we agree that a cosmic ray can be reliably identified if the minor axis of the 2nd moments is < about 0.85 pixel2, and if the Kron (or basic aperture mag, if needed) is < about -8.5 instrumental mags. For the star/galaxy separation issue, we divided this into two questions: (1) what is the a-priori probability that such-and-such an object is a PSF (in the absence of any knowledge about other kinds of shapes)? (2) given (1) and the distribution of galaxy shapes and sizes, what is the probability that a given source is a star or a galaxy? It looks like (1) can be defined quite well on the basis of the PSF - Kron magnitudes. (2) is much harder, depends on the distribution of galaxy shapes, and is probably outside of the scope of the IPP processing, and more relevant to the morphology analysis client. We can, however, make some initial estimates based on the comparison of GPC1 and the SDSS analysis of the Stripe 82 deep stack. Rather than worrying about getting (2) exactly right, we should be instead be answering the question: for which objects should we perform some further extended source analysis? Not all objects in this class need to have a 100% probability of being a galaxy. The Stripe 82 comparison gives us some guidance, for a specific ratio of stars / galaxies, or equivalently at a specific Galactic latitude.

Relatedly, we have had reports that the psphot version that is used for the 'static sky' analysis (psphotStack) is not producing all of the outputs expected, including the extended source fits and the radial aperture photometry. I tracked down some configuration and software errors that were responsible these issues (failure to initialize the aperture sizes, default S/N limit for petrosians set quite high, etc). The more significant issue is the failure to generate all of the fits. In trying to understand this, I discovered that the convolution to a target PSF was resulting in a somewhat uglier output PSF than expected, and this was making the source detections go haywire. Although I can get better convolved stacks by tweaking the kernel options, I not entirely satisfied by the result. I've asked Mark Huber to help me with this by seeing how hotpants behaves on some specific image pair covering a wide range of seeing differences.

Serge Chastel

  • IPP:
    • databases questions: all optimization of nebulous attempts failed (ippdb02, ipp022, tried also locally);
    • pretended to be czar on Wednesday and Thursday;
    • wrote scripts to set hosts off on all pantasks servers. Wrote also a script to check hosts status on the different servers but it needs to be improved (since it seems the information given by pantasks server is not always reliable).
  • PSPS:
    • played a bit with selenium and phpunit.
  • MOPS:
    • czar (with Larry)

Heather Flewelling

Roy Henderson

  • PSPS
    • resumed batch creation, but not loading until merge/flip/copy is complete
    • meeting with Jim/Gene about stacks. Consequent schema changes
    • tracked the source of loading duplicate frames: datastore/DXLayer logic
  • IPP
    • czartool
      • added option to make a useful plot of a single day
      • removed the zeros from labels table (Bill's suggestion)
      • work on a 'daily report' for Ken (not finished yet)
      • fair bit of work on a 'rate of processing' plot
    • ippToPsps
      • added database access to ippToPsps: needed for stack stuff
      • slowly populating fields for PSPS stacks
      • fun with fits binary table 'vectors'
  • Mailing lists
    • figured out how to add new lists (code and Db changes)
    • wrote a backup script to dump Db etc (needed as everything is on neverland just now)
    • new sts list

Bill Sweeney

  • Made several improvements to the handling of errors in update jobs queued by the postage stamp server. (The errors are caused by data that has been lost on ipp020). The update system is now much more capable of recovering for errors without manual intervention. This is an extremely important milestone for Bill as these problems have been an huge time sink and source of stress.
  • Investigated the source of slow download performance seen by MOPS from the ipp data store. Two sources were found. First the perl module used for downloading has some inefficiencies. Secondly the web proxy can be a bottleneck. MOPS will now bypass the proxy and access the data store directly.
  • Implemented ordering by priority for diff, destreak, and stack stages.
  • Fixed bug where magicRuns were queued even though a diff run failed to generate any good quality skycells
  • Processed 3 nights X 36 exposures from CFHT-MEGACAM through the IPP for the MOPS team. This went remarkably smootly given that we have not processed data from that instrument lately. Unfortuntely astrometry failed on several chips per exposure so some data was lost.
  • Wrote a program to examine the nebulous database to discover all components that can never be updated because the original run information was lost on ipp020.
  • Wrote a program to find all of the data listed on the datastore that is unavailable because data on ipp020 was s lost.
  • Spent half a day dealing with the fallout due to the crash of the mysql database backing up the postage stamp server and data store.
  • Updated pstamp-test system (ipp049) published CFHT detections for MOPS there while the DB was recovered.
  • Processing czar on Friday. Helped out acting czar on other days. Did some label and list manipulations to allow the OSS diffs to jump ahead in the queues.

Chris Waters

  • Linearity:
    • tested that input exposures are stable over time (~4 months).
    • added smoothing feature to remove bad row corrections.
    • bug fixes in code and math.
  • Nebulous: migrated ipp020 to ipp020X, and readded a fresh volume after RAID failure.
  • Scheduled flights/hotels for JHU liaison trip.