IPP Progress Report for the week 2011.03.28 - 2011.04.01

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

Heather Flewelling

Roy Henderson

The bulk of my work this week has been big changes in ippToPsps to accommodate the new stacks provided by Gene.

In order to facilitate an easy transformation of the new stack FITS files to the format required by PSPS, it became clear that using a database-in-the-middle was the only sensible way forward. The alternative was to write hundreds of lines of C to essentially 'query' the contents of FITS tables, therefore re-inventing what databases already do (this was less of an issue with detections, which have a much more direct mapping from IPP to PSPS fields). So, I opted to use STILTS, which is a Java project for manipulating tabular astronomical data (STILTS underlies the TOPCAT package, which I use on the PSVO project). STILTS allows you to read and write FITS tables, while also enabling them to be written to and read from any database, making it a perfect fit for this work.

I chose to use Jython for the coding, which is a Java implementation of Python that enables the use of Java classes as well as almost all Python modules. In a fraction of the lines of code it takes in C, I am able to import IPP FITS files, store them in a MySQL database, create new PSPS-shaped tables (as defined by the PSPS schema), populate those tables through SQL inserts/updates, then export them all to an output FITS file. This way, FITS is used solely for import and export, not as a format to be queried, for which it is inflexible and cumbersome.

The steps to this have been as follows. So far, only the init and stack batch types have been implemented, with detections to follow when time permits.

• changed the pspsSchemaToXml script to output PSPS table definitions in the VO-standard VOTable format
• the VOTable files are then used by STILTS to generate empty PSPS MySQL tables
• for init tables, the data section of the VOTable file is populated with known values (survey IDs, filter IDs etc) and STILTS is able to create populated tables in the database from these, then export them to FITS as 'IN batches'
• IPP fits files are imported into MySQL tables using STILTS, and indexes added to all IPP detection ID columns to improve speed
• it was necessary to write a parser to import the IPP FITS primary headers, as STILTS ignores primary headers
• wrote all the SQL inserts and updates necessary to populate the PSPS tables using the IPP tables
• STILTS then exports all PSPS tables to output FITS format

Bill Sweeney

Chris Waters