IPP Progress Report for the week 2010.10.25 - 2010.10.29

(Up to IPP Progress Reports)

Eugene Magnier

After a couple of false starts and delays, we have finally updated the operational IPP to the latest version.

The regular processing is now being performed with the IPP tag ipp-20101029 (the Black Friday tag...). This can be found in our svn tree at: tags/ipp-20101029

This update incorporates a number of changes that have been in the works and in testing since the meeting at Belfast. Here is a summary of the most important changes and improvements:

  • the Star likelihood measurement and the Cosmic Ray detection are improved, see the discussion at: StarGalaxySeparation
  • Cosmic Rays are now masked in the chip-stage output images
  • Some improvements to both the DUAL and SINGLE convolution analysis, discussed at the following link (more discussion to be added soon): ppSubModifications_201009
  • Stack masking now includes information about the cause of masked pixels, as well as information about suspicious pixels: Stack
  • Fixes to the bright star masking: The size of the bright star circles was OK for very bright stars (-23 inst mags), but too small for stars at or just beyond the saturation limit. This is now fixed. Also, the w-band images used too small of a zero point for the over all scaling.
  • STACK-STACK processing now uses SINGLE convolution of the template instead of DUAL convolution
  • addressed a couple of low-level bugs that caused errors in the calculation of centroids or screwed up the peak positions in some chips
  • for faint sources, where we do not perform a full non-linear fit for the position, we are now reporting the position and position errors based on the centroids rather than on the peak in the detection image. With the fix above, this gives much more reliable values for the position and errors on these faint sources.
  • for forced photometry, we are now keeping the supplied position (the code had been letting the position be reset by the pixel analysis, which is definitely wrong).

This release includes some fixes to psphotStack, which is used for the 'static sky / deep stack' photometry, but there are a few outstanding concerns that have not yet been addressed. This release also does not address the STS desire for forced photometry in the diff images of the input image sources.

Also note that MOPS will continue to receive the old format tables until they are ready; the detections are being generated at the diff stage with the new format (PS1_DV2), but the publication process is producing the old format.

Finally, this is NOT the final IPP version release before the Great Reprocessing. Aside from the issues listed above, there are some other things to be addressed. We will be producing sample re-processing of MD04 and SAS data, among others, for feedback. I suspect it will be 2 - 3 weeks before everything else that needs to be addressed before the Great Reprocessing can be addressed.

Serge Chastel

  • ppTranslate code update for PS1_DV2
  • New option for ppMops: -version
  • IPP-MOPS ICD update (with PS1_DV2)
  • Replication of Nebulous: now replicated on ipp022

Heather Flewelling

  • vacation

Roy Henderson

  • PSPS
    • 2nd interviews with Jason and Thomas and follow-on discussions
    • usual 3PI loading (8650 more frames are now loaded and ready for merge)
    • started work on PSPS VO Toolbox in Java
      • wrote a SOAP client class to connect and query PSPS Db
      • investigated the VO tools (VOPlot, Topcat etc), how to integrate, re-use classes etc
      • integrated soap client with a simple plotter, able to pull stuff from PSPS Db and plot easily
      • set up a project using Eclipse. Able to use starlink classes
  • IPP
    • more roboczar work. now more reliable in it's 'warning' emails

Bill Sweeney

  • Finished implementing and testing changes to destreak stage cleanup and updated. Integrated into working system.
  • Wrote a program to recover the destreak statistics into the database.
  • Spent some time debugging damaged home directory problem. Gavin ran raid verification and the problem has not yet reoccurred.
  • Debugged and fixed some postage stamp server bugs caused by the destreak re-work.
  • gathered some data for the outreach program and copied it to ipp022. (should have scripted this better the first time.)
  • processing czar one day.
  • Fixed some more update dependency checking failures. Add some modes to magicdstool to list outputs of runs.
  • Debugged a distribution failure. It turns out the system has been repeatedly trying to post a fileset from data from July that was lost on ipp020. If the fault isn't on czartool we don't see it.
  • Extracted code from automate_stacks to create a script to queue unmagicked OSS diffs.
  • Studied the publish system and configured it to handle queueing unmagicked OSS diffs.
  • Fixed some cluster problems that happened over the weekend.

Chris Waters

  • Trip to JHU and a day of travel. Summarized trip results and questions and emailed answers around to ps-ipp-dev and panstarrs@jhu.
  • Unsuccessfully searched for missing burntool tables. Will need to redo this before reprocessing.
  • Added reduction classes to the difftool commands in survey.pro.
  • Designed a set of jobs that need to be done by an in-progress neb-fsck program. These jobs are needed to ensure that the database and the disk match. Correcting these inconsistencies is needed for the update code to run without intervention.