IPP Progress Report for the week 2010.06.07 - 2010.06.11

(Up to IPP Progress Reports)

Eugene Magnier

I've been largely focused on generating a clean reference photometry and astrometry database for MD04. Progress was slow because I spent most of the time discovering and fixing low-level bugs with the dvo database manipulation tools (dvomerge and photdbc, the latter of which had not yet been used very much). This was a good exercise, however, as if gives a good demonstration set to prove that all of these steps and tools are working. Here are the steps I followed (I am putting the actual commands on the wiki):

First, I generated a griz database for MD04 from the Demo Month smf files for which the data was magicked. I discovered that we have not magicked (nor distributed) any of the y-band data for MD04. This is because there is no reference stack in y-band. I used some of the y-band smf files to generate a separate y-band dvo database. I ran relphot and relastro on both with several iterations to calculate good internal magnitudes and positions. I then used photdbc to extract a high-quality database for both the griz and y versions: this tool makes a copy of a dvo database, potentially restricting the data used in the copy. In this case, I limited reference database to the objects with more than 40 detections in griz (and 10 in y) and at least one of the average magnitudes needed to have an error < 0.025 mags. I then used dvomerge to combine these two reference databases.

To tie down the zero point, I made a copy of this merged database and used addstar to incorporate some of the SDSS data from this field. I then extracted the photometry in the internal average PS1 system matched to the corresponding SDSS photometry and determined zero points. I reset the master databases to use these SDSS-tied zero points and regenerated the high-quality reference database. Note that for y-band, the zero point is crude: it is based on setting stars with i-z = 0.0 to have z-y = 0.0 (after i and z were corrected into the SDSS system). I did NOT apply any color terms at this point to determine the SDSS transformation, which means the zero points are only set for the mean stellar color.

Finally, I merged the data for a wide region around MD04 from synthetic grizy database into this high-quality reference, using the magnitudes and positions of PS1 measurements where an object was present in the high-quality database, but accepting the synthetic data if no object was present. This last step allows us to perform astrometry with this database even on the images (and parts of images) that stick out beyond the main portion of the MD04 region. This reference database is now installed on the cluster, and I have begun the processing of the MD04 images using this new database as a reference.

Serge Chastel

  • read various documentation
  • currently implementing histogram visualization in ippMonitor

Heather Flewelling

  • minidvodbs
    • debugging/testing of minidvodbs
    • successful merge of 56 minidvodbs
    • sent this 56/merged dvodb to Roy for further testing (it passed)
    • started minidvodb catalogs of ThreePi.nightlyscience data
  • further investigations of isp/gpc1 skybrightness
    • couldn't find evidence of isp 'chevrons' or other wierdness during non-bright times, but there are not many images that I think would have them that I could look at.
  • ifaps1
    • answering ifaps1 questions as they come up
    • updated ipp build
    • updated ippdb with the build
    • did not restart the download - at 95% full. Planning to deal with this next week.
  • helped Serge get started with various IPP things (ippmonitor, ipp, showed him where coffee was...)

Roy Henderson

  • PSPS
    • BIGINTs slipping through as objID and failing at load: fixed
    • Enlisted Gavin to help with wiki permissions thing. Finally fixed
    • Neatened up and committed fitsConverter code for new DXLayer
    • Problems with interconnectivity within MHPCC, i.e. between IPP and PSPS clusters: Gavin to the rescue again.
  • IPP
    • Big changes to support new DXLayer work by Conrad: ditched the idea of 'jobs', now publishing 'batches'
    • Now using 'dist_group' from IPP against PSPS init batch data to set surveyID
    • Improved handling of 'init' batch creation in run script
    • Helped Heather confirm that the merged DVO Dbs looked ok by running them through ippToPsps
    • Looked into, and fixed, problems with exposure times: a) lack of precision, b) was not using mid-exposure time for detections

Paul Price

  • Fixed SEGV in normalisation of darks: chips weren't being mosaicked due to recent change.
  • Completed rework of ppMops to generate SMF-like format
  • Optimised ppStack for stacks of many images: create single target PSF image for all inputs to share
  • Coded ppBackground to restore background of an image, given pattern and background model

Bill Sweeney

  • Postage stamp server testing.
  • Still tracking down problems with update processing uncovered by users postage stamp requests. There are some time consuming problems, but the good news is most requests are working fine. 1.2 Million postage stamps have been generated so far.
  • In my two days as processing czar I did a comprehensive analysis of the currently unfinished processing. Results are on this wiki page.
  • added some code to pstamptool and disttool to facilitate queuing data for cleanup.

Chris Waters

  • Various bugfixes to get simtest to complete properly. Merged into the trunk.
  • Ran multiple simtest runs to test the functionality and fix stack associations and warp/stack/diff summary stages.
  • Added MOPS detectability server datastore to the postage stamp server database so they can be automatically processed.