IPP Progress Report for the week 2011.03.28 - 2011.04.01

(Up to IPP Progress Reports)

Eugene Magnier

I spent most of the week working on the reference photometry database. My main focus was on the MD field database, to confirm the quality of the relative photometry analysis and the zero point measurement. As mentioned last week, I needed to rebuild this database with only a single version of each exposure. I also modified relphot to exclude suspected galaxies from the calibration analysis. Including these fixes, I still found surprisingly high zero point variations: exposures from a single night tended to be in good agreement, but those from nearby nights could be off by as much as 2% in either direction. I would not be surprised at long-term variations, but variations on 2-3 day time scales do not seem reasonable.

Looking at this further, I discovered that the measured zero points trended with the image quality. I believe this is a reflection of a PSF-Aperture correction which uses too small of an aperture: it is still sensitive to the seeing variations. Unfortunately, the dvo database currently does not include the mean FWHM for each chip, so studying this trend was not simple. I created a tool to ingest the fwhm values into DVO, and to define the median exposure FWHM from those values. I was then able to measure the trend of zero point offset vs median FWHM. Fitting a single quadratic curve to all zero point variations from all MD field exposures in a single filter brings the night-to-night scatter into a very respectable range: 1.0-1.5%. The variations that remain (outside of obvious clouds) are somewhat long term (~50 day structures), and seem to agree between the different filters. This is not yet conclusive, but suggests that I now detecting the real system variations (probably atmospheric). Also important: the quadratic fit to all four filters griz agree very well: this makes it reasonable to treat this effect as a color-independent trend. (The MD y-band measurements are somewhat more scattered, and need some further examination).

I am continuing now to work on the assessment of the photometry in the 3pi reference database.

Serge Chastel

  • MOPS czar
  • IPP czar
  • gpc1 replication now ok
  • hemocrat development
  • quality in IPP for MOPS

Heather Flewelling

  • 2 days vacation
  • 2 days czar -- uneventful - had to revert a few corrupted files
  • restarted addstar minidvodb merging on wed. (also finding space to clean up on ipp004)
  • ippMonitor - fixed plotting for isp
  • finished MD09.jtrp, started MD10.jtrp
  • investigated john's list of missing MD stacks
    • some lack enough exposure to make a stack (we never downloaded from summit)
    • ~.5 were cleaned up before he could download
    • ~.5 were not on the original list
  • transferred MD04 deepstack to ifaps1 for Tom.
  • started working on addstuff modifications for stacks

Roy Henderson

The bulk of my work this week has been big changes in ippToPsps to accommodate the new stacks provided by Gene.

In order to facilitate an easy transformation of the new stack FITS files to the format required by PSPS, it became clear that using a database-in-the-middle was the only sensible way forward. The alternative was to write hundreds of lines of C to essentially 'query' the contents of FITS tables, therefore re-inventing what databases already do (this was less of an issue with detections, which have a much more direct mapping from IPP to PSPS fields). So, I opted to use STILTS, which is a Java project for manipulating tabular astronomical data (STILTS underlies the TOPCAT package, which I use on the PSVO project). STILTS allows you to read and write FITS tables, while also enabling them to be written to and read from any database, making it a perfect fit for this work.

I chose to use Jython for the coding, which is a Java implementation of Python that enables the use of Java classes as well as almost all Python modules. In a fraction of the lines of code it takes in C, I am able to import IPP FITS files, store them in a MySQL database, create new PSPS-shaped tables (as defined by the PSPS schema), populate those tables through SQL inserts/updates, then export them all to an output FITS file. This way, FITS is used solely for import and export, not as a format to be queried, for which it is inflexible and cumbersome.

The steps to this have been as follows. So far, only the init and stack batch types have been implemented, with detections to follow when time permits.

  • changed the pspsSchemaToXml script to output PSPS table definitions in the VO-standard VOTable format
  • the VOTable files are then used by STILTS to generate empty PSPS MySQL tables
  • for init tables, the data section of the VOTable file is populated with known values (survey IDs, filter IDs etc) and STILTS is able to create populated tables in the database from these, then export them to FITS as 'IN batches'
  • IPP fits files are imported into MySQL tables using STILTS, and indexes added to all IPP detection ID columns to improve speed
  • it was necessary to write a parser to import the IPP FITS primary headers, as STILTS ignores primary headers
  • wrote all the SQL inserts and updates necessary to populate the PSPS tables using the IPP tables
  • STILTS then exports all PSPS tables to output FITS format

Bill Sweeney

  • This week was spent analyzing the latest GPC1 static mask. This involved
    • selecting and processing a set of test exposures to run though the IPP up to magic streak detection
    • writing programs to extract the clusters that magic detects, transforming them to chip coordinates and then making images for each chip where the pixel value is the number of times that a particular pixel triggered magic.
    • Heather took these data and updated the static mask.
    • Ran the exposures though the system again and verified that the magic results were improved
  • Also spent some time trying to finish up the 331 ThreePi?.rerun exposures for Bertrand. This didn't quite get done this week.

Chris Waters

  • Update fringe correction procedure to require more points per binned sample, as well as using a clipped mean and standard deviation. This removes points that were being improperly weighted and skewing the final fringe value.
  • Constructed new dark for OTA47 using data taken after the camera voltage change that improved corner glows (2010-09-13). Confirmed that this improved this chip and registered a new darktest detrend that includes this OTA along with the OTAs of the previous darktest.
  • Upgraded ISP database to use summit_id (details: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_CzarLog_20110328).
  • Updated deteff CDF values in dqstats. These values are good for gri, but need further updates for yzw.
  • Identified SQL JOIN bug in faketool that was clogging up science processing. No idea why this suddenly became a problem, but likely due to the growing size of the chipProcessedImfile.
  • Studied astrometry bug first identified in STS data. This appears to be a property of the 2-phase devices only, with further details at: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC_BrightStarAstrometry_Issue. Waiting for microtest data to help identify a solution to this bug.
  • Tracked odd background pattern in OTA43/cell54 to a bad overscan issue. Applying PATTERN.CELL fixes most of the offset, but ~70 columns of pixels will need to be masked on the static mask.