IPP Progress Report for the week 2011.03.28 - 2011.04.01

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• MOPS czar
• IPP czar
• gpc1 replication now ok
• hemocrat development
• quality in IPP for MOPS

Heather Flewelling

• 2 days vacation
• 2 days czar -- uneventful - had to revert a few corrupted files
• restarted addstar minidvodb merging on wed. (also finding space to clean up on ipp004)
• ippMonitor - fixed plotting for isp
• finished MD09.jtrp, started MD10.jtrp
• investigated john's list of missing MD stacks
  • some lack enough exposure to make a stack (we never downloaded from summit)
  • ~.5 were cleaned up before he could download
  • ~.5 were not on the original list
• transferred MD04 deepstack to ifaps1 for Tom.
• started working on addstuff modifications for stacks

Roy Henderson

The bulk of my work this week has been big changes in ippToPsps to accommodate the new stacks provided by Gene.

In order to facilitate an easy transformation of the new stack FITS files to the format required by PSPS, it became clear that using a database-in-the-middle was the only sensible way forward. The alternative was to write hundreds of lines of C to essentially 'query' the contents of FITS tables, therefore re-inventing what databases already do (this was less of an issue with detections, which have a much more direct mapping from IPP to PSPS fields). So, I opted to use STILTS, which is a Java project for manipulating tabular astronomical data (STILTS underlies the TOPCAT package, which I use on the PSVO project). STILTS allows you to read and write FITS tables, while also enabling them to be written to and read from any database, making it a perfect fit for this work.

I chose to use Jython for the coding, which is a Java implementation of Python that enables the use of Java classes as well as almost all Python modules. In a fraction of the lines of code it takes in C, I am able to import IPP FITS files, store them in a MySQL database, create new PSPS-shaped tables (as defined by the PSPS schema), populate those tables through SQL inserts/updates, then export them all to an output FITS file. This way, FITS is used solely for import and export, not as a format to be queried, for which it is inflexible and cumbersome.

The steps to this have been as follows. So far, only the init and stack batch types have been implemented, with detections to follow when time permits.

• changed the pspsSchemaToXml script to output PSPS table definitions in the VO-standard VOTable format
• the VOTable files are then used by STILTS to generate empty PSPS MySQL tables
• for init tables, the data section of the VOTable file is populated with known values (survey IDs, filter IDs etc) and STILTS is able to create populated tables in the database from these, then export them to FITS as 'IN batches'
• IPP fits files are imported into MySQL tables using STILTS, and indexes added to all IPP detection ID columns to improve speed
• it was necessary to write a parser to import the IPP FITS primary headers, as STILTS ignores primary headers
• wrote all the SQL inserts and updates necessary to populate the PSPS tables using the IPP tables
• STILTS then exports all PSPS tables to output FITS format

Bill Sweeney

• This week was spent analyzing the latest GPC1 static mask. This involved
  • selecting and processing a set of test exposures to run though the IPP up to magic streak detection
  • writing programs to extract the clusters that magic detects, transforming them to chip coordinates and then making images for each chip where the pixel value is the number of times that a particular pixel triggered magic.
  • Heather took these data and updated the static mask.
  • Ran the exposures though the system again and verified that the magic results were improved
• Also spent some time trying to finish up the 331 ThreePi.rerun exposures for Bertrand. This didn't quite get done this week.

Chris Waters

• Update fringe correction procedure to require more points per binned sample, as well as using a clipped mean and standard deviation. This removes points that were being improperly weighted and skewing the final fringe value.
• Constructed new dark for OTA47 using data taken after the camera voltage change that improved corner glows (2010-09-13). Confirmed that this improved this chip and registered a new darktest detrend that includes this OTA along with the OTAs of the previous darktest.
• Upgraded ISP database to use summit_id (details: ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_CzarLog_20110328).
• Updated deteff CDF values in dqstats. These values are good for gri, but need further updates for yzw.
• Identified SQL JOIN bug in faketool that was clogging up science processing. No idea why this suddenly became a problem, but likely due to the growing size of the chipProcessedImfile.
• Studied astrometry bug first identified in STS data. This appears to be a property of the 2-phase devices only, with further details at: ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC_BrightStarAstrometry_Issue. Waiting for microtest data to help identify a solution to this bug.
• Tracked odd background pattern in OTA43/cell54 to a bad overscan issue. Applying PATTERN.CELL fixes most of the offset, but ~70 columns of pixels will need to be masked on the static mask.