IPP Progress Report for the week YYYY.MM.DD - YYYY.MM.DD

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

  • IPP
    • Set up the ipp framework on mops14 for synthetics analysis. All necessary modifications to the original IPP are in ipp@mops14:~/dev/ipp_modifications. I also have detailed notes on the installation process (all the files and database operations which are required to have a running ipp): now I just need to write them down in a wiki. On Friday I had been able to run the full processing (till mops diff products). Two concerns though: (1) there is still a step which is necessary to be manually run between chip and cam stages; (2) Processing a quad gave me 3 products instead of 2. I have to investigate why.
    • I haven't been able to install scipy with Python 2.6.5 which is by default on the system so I installed 2.7 for the ipp account.
    • Started to set up the ipp for megacam images processing. I created a dedicated account for that: ippmc@mops14. I successfully got registration running (thanks to Heather and her 1504 ticket). When I tried to run MegaCam? images through the IPP though, it stopped at chip stage with weird half empty logs. I'm stuck at this point now.
    • I also created a dedicated account that will be used for Subaru image processing: ippsu@mops14
  • MOPS
    • I'm thinking of using what I did about 2 years ago, that is, Condor for the IPP to replace pantasks. Two reasons for that: (1) I will not have to modify the high-level IPP scripts (e.g. chip_imfile.pl) which are necessary to process the synthetics. (2) Modifications to the regular processing will likely be easier (through the use of DAGs) if we want to add or perform some stages differently (e.g. Steve Hartung's diff).

Heather Flewelling

  • gene found slice18 one of the minidvodbs was missing data due to corrupted skytable.fits. I made a new minidvodb and 'hid' the old one
  • sent serge a 'hack' for getting non-datastore stuff into databases
  • MD09 queued
  • investigation of batchid = -999
  • figured out that the holey objects in psps come from holey batches on our side
  • attempted to fix czarstuff to monitor ipptopsps - has some problems due to gene's change to skychunk
  • w band dvo : first attempt failed due to config errors: (W.20130226), next set is working (W.20130227)
  • broke mysql replication while doing ssp datastore stuff - warned serge
  • ssp datastore now being downloaded!
  • various monitoring of ipptopsps
  • updated http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/LAPintoPSPS
  • various queries of psps database (to see if what we loaded looks ok)
  • cleanup of space on ipp004/ipp005 - we were running low
  • various work with CA/ROTSE

Mark Huber

Bill Sweeney

  • Spent most of this week putting the finishing touches on the software for tracking ipp releases. One output of this will be a web site that may be used to access camera stage detection files (smfs). The difference between this and the standard distribution system is that it does not require second copies of the data to be stored on the cluster and thus can be used "permanantly".
  • Corresponded with Eddie Schlafly, one of the primary consumers of the new smf retrieval system to make sure that it will meet the needs of the ubercal project.
  • In preparation for populating the release management tables investigated the frequency of duplicate LAP runs in the current release. Wrote scripts to hide the extra results.
  • Designed a method to manage the completion of lap processing for a region. This includes a new lightweight table called lapGroup. This will be used
    • to signal that a set of 5 lap runs for a given projection cell have completed
    • that it is time to queue the staticsky runs
    • and that staticskyRuns have been queued for those skycells.
  • Worked a bit on the ipp side of the new PSI postage stamp web service.
  • two days as processing czar
  • consulted with Chris as to how to fix problem where lap was blocked because cleanup was not running.

Chris Waters

  • Identified issue with trail fit parameters. Something is incorrect in the fitting algorithm, which results in the trail length for all objects to be driven to infinity. Discussed this result with MOPS, and Peter and Eugene will work together to find the algorithm error and fix it.
  • Looked into the M31 OTA offset issue, and updated wiki page here. This looks like an issue in the dark model for some OTAs. I also demonstrated that the camera stage background restored method seems to agree well with images where no background has been removed.
  • Began processing for David Thilker's large galaxy list. I'm trying to do this incrementally, so as not to fill up the disks (there are ~3800 exposures for the primary target list).
  • Worked on the excess detection/incorrect variance issue detailed here. This issue seems to arise due to the "additional variance" term that is calculated during stacking. This additional variance seems to de-weight all exposures except one, resulting in the final stack having a calculated variance image that assumes all inputs contributed equally, but with the data image showing only the scaled single input. This results in the image being noisier than the variance map, and the photometry identifies all the image peaks as significant because of this.
  • Dropped LAP stacks that can not complete.