= IPP to PSPS interface: {{{ippToPsps}}} = [[PageOutline]] {{{ippToPsps}}} is the interface between IPP and PSPS. It creates FITS files generated from a multitude of IPP data-sources, then publishes them to a datastore in the form of ''batches''. On the PSPS side, the {{{DXLayer}}} polls the datastore, collects batches when they become available, then converts the contents to {{{csv}}} files before sending them on to SQL Server loader software, which ''merges'' them into the PSPS database. It is intended that the binary tables in the FITS files generated by {{{ippToPsps}}} match the PSPS database schemas perfectly, the consequence being that any alterations to the PSPS database schema will only affect {{{ippToPsps}}} code, and not the {{{DXLayer}}}. A certain amount of data validation is performed by {{{ippToPsps}}} before publication, with more validation occurring at the loading and merge stages on the PSPS side. Simultaneously to loading data, {{{ippToPsps}}} polls PSPS to inquire after the status of the batches it has loaded. Batches that failed can then be reloaded. The status of every batche is maintained in a MySQL database local to {{{ippToPsps}}}. The outputs of {{{ippToPsps}}} are referred to as 'batches', and are detailed below. ||'''Batch name''' || '''PSPS name''' || '''Description''' || '''IPP Source''' || || [wiki:ippToPsps_Initialization Initialization] || IN || metadata relating to other batches, eg filter ID, survey ID etc || generated from XML config || || [wiki:ippToPsps_Detections Detections] || P2 || single exposure detections || generated from one {{{smf}}} file per exposure plus associated DVO database || || [wiki:ippToPsps_Stack Stack] || ST || stack image detections || one FITS file generated per IPP {{{cmf}}}, which contains data for one filter on one skycell || || [wiki:ippToPsps_Difference Difference] || ? || difference image detections|| generated from one {{{cmf}}} file per skycell per exposure || [[BR]] = Configuration = Due to the potential for changes in both input and output for {{{ippToPsps}}}, rather than hard-coding table descriptions, the code is heavily configurable. As such, the tables descriptions are stored as [http://www.ivoa.net/Documents/VOTable/20040811/REC-VOTable-1.1-20040811.html VOTable] files (for [http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/browser/trunk/ippToPsps/config/detection/tables.vot example]), which are a standard of the [http://www.ivoa.net/ IVOA]. Because VOTables are an XML format, they both human and machine readable, expandable and self-describing. These VOTables are generated directly from the PSPS schema using a [http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/browser/trunk/ippToPsps/perl/pspsSchema2xml.pl script], so that any changes to the schema can be easily passed-along to {{{ippToPsps}}}. Additionally, a more general configuration file is used to detail the names of databases, location of output files, datastore settings etc. This can be seen [http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/browser/trunk/ippToPsps/jython/config.xml here] = Architecture = {{{ippToPsps}}} is written in [http://www.jython.org/ Jython], this is in part to take full advantage of the [http://www.star.bris.ac.uk/~mbt/stilts/ STILTS] package, which enables very fast and efficient processing of astronomical catalog data tables. Since it supports FITS, VOTable, and SQL, it is a perfect fit for this project. It is also software maintained elsewhere, reducing this burden for us. == Design == Each batch type (detection, stack, init etc) is its own class, all of which inherit from the Batch base-class, which handles features common to all batches, such as creating and opening output FITS files, connecting to the GPC1 database, connecting with the DVO database etc. This keeps duplicated code at a minimum. The Batch class is an abstract class, i.e. it should not, and cannot, be instantiated. = Running = Notes on how to run ippToPsps can be found on the individual batch-type pages, linked to at the top of this page. == Deletion policy == When loading, two copies of the data exist: the batches on the datastore and the original files on disk. The deletion policy at present is: * when a batch has been loaded to the ODM and has a status of 'merge worthy', then the copy on the datastore is removed * when a batch has been successfully merged into the PSPS database, the copy on local disk is deleted The logic for this is that errors may occur during the merge phase and it is useful to have local copies of offending batches for debugging purposes. = Unit testing = Unit tests exist for all batch types in the {{{unitTests}}} folder. These tests confirm that data integrity is maintained after code changes have been made. All batch tests can be run like this {{{ cd unitTests ./runAllTests.sh }}} Or, individual test like this, for detections: {{{ cd unitTests/detectionTest ./run.sh }}} All test create batches in a 'test' mode, meaning, a current date will not be used and, in the case of detections, only chip 33 (instead of all 60) will be processed. The resultant FITS files are then compared (using {{{diff}}}) with FITS files known to be correct. The user is informed if they are identical or if they differ. = Recovery system design = Currently, the IPP to PSPS interface is a 'one-way' system. Batches are created by {{{ippToPsps}}} and posted on an IPP instance of the datastore. These batches are collected by the {{{DXLayer}}} on the PSPS side and sent on to the ODM. The IPP urgently requires some feedback from PSPS to determine which batches have succeeded and which have failed (and why they failed). With this information, data can be either deleted or regenerated accordingly. This is important simply because, with such large data volumes, we cannot afford the high levels of redundancy currently in place. At present, for a given batch, the following copies exist within the pipeline: - a copy exists on the IPP cluster after generation by ippToPsps program - a copy exists on the IPP datastore after publication by ippToPsps - the {{{DXLayer}}} retains a copy after it has sent the csv version to the ODM - the {{{DXLayer}}} also keeps a copy of these (larger) csv files We therefore need to quickly implement the basic framework of a feedback loop so that the IPP can quickly learn if a given batch has been successfully merged into the PSPS database or not. This will enable it to safely delete the data files and remove the copy from the datastore. This will also form the basis for a more comprehensive recovery system, to be developed at a future date. == Previous design == {{{ ............................. . ___________ . . | | . ---------------------------------------| datastore | . | . |___________| . .......|........................... . /|\ . . ____\|/_____ ___________ . . ____|____ _____ . . | | | | . . | | | | . . | ippToPsps |----->| datastore |-------------->| DXLayer |<---->| ODM | . . |___________| |___________| . . |_________| |_____| . . . . . ................................... .............................. IPP PSPS }}} Previously, Conrad and I had discussed a design whereby a second datastore instance would be utilized, this time on the PSPS cluster. The {{{DXLayer}}} would act as the 'middle-man', polling the ODM for updates on loading progress, then posting the results on the PSPS datastore for the IPP to consume. Polling this, {{{ippToPsps}}} could acquire a list of batches it knows are safe to be discarded. Simultaneously, the {{{DXLayer}}} could delete its copies of the same redundant data. The update placed on the PSPS datastore could take the form of an XML file. At first this would simply detail those files it is safe to delete, but could evolve into a more complex recovery report, i.e. which batches failed, and what is required to be done by the IPP. == New design == {{{ ------------------------------------------------------------ | | ........|........................... .......................|..... . ____\|/_____ ___________ . . _________ __|__ . . | | | | . . | | | | . . | ippToPsps |----->| datastore |-------------->| DXLayer |----->| ODM | . . |___________| |___________| . . |_________| |_____| . . . . . ................................... .............................. IPP PSPS }}} Instead of creating a new datastore instance within PSPS and using the {{{DXLayer}}} as communication layer between the ODM and the IPP, we propose that the {{{DXLayer}}} forms no part of the feedback system. It should be simplified such that it only facilitates loading, i.e. polling the IPP datastore for new data, converting it to csv files then sending these on to the ODM. Instead, to complete the circle, the {{{ippToPsps}}} code will poll the ODM directly, bypassing the {{{DXLayer}}} altogether. This also forms the basis of a full recovery system as, at a later date, {{{ippToPsps}}} can be coded to respond intelligently to the myriad of errors that may occur within the ODM. The {{{DXLayer}}} need know nothing of the how or why a certain batch is being submitted by the IPP, it should just grab it, convert it and pass it along to the ODM. This design would therefore mean simplifying a major PSPS component, the {{{DXLayer}}}, but rather than waste the code already written, it would be taken and used within {{{ippToPsps}}} (for example, the ODM polling scripts). We would simply be shifting responsibility over from PSPS to IPP. Over parts could be dropped completely. For example, since {{{ippToPsps}}} will (soon) keep a record of all the jobs and corresponding exposure IDs in the IPP database, it is unnecessary for this information to be duplicated by the {{{DXLayer}}}, which currently has its own local database for this information. The question remains of what should be done with the copies of the data currently retained by the {{{DXLayer}}}? The options are that it can either be deleted automatically after a defined amount of time, or the IPP can send a list of batches it is safe to delete through the datastore, or perhaps the {{{DXLayer}}} should not retain files at all. Since it can quickly and easily acquire data from the IPP datastore anyway, it is probably unnecessary for it to hold any copies. === Advantages over previous design === - no need for second datastore (not a big overhead, but it would require additional systems administration in an already complicated system) - no need to define new XML standard that incorporates the whole array of recovery options - no need for the {{{DXLayer}}} to poll the ODM - no need for the {{{DXLayer}}} to have a database to log the batches (already done on the IPP side) - no need for the {{{DXLayer}}} to keep data at all? = links = [http://datastore.ipp.ifa.hawaii.edu/PSPS_test Datastore test area for PSPS on Maui][[BR]] [http://datastore.ipp.ifa.hawaii.edu/PSPS_JHU Datastore test area for PSPS at JHU][[BR]] [browser:trunk/ippToPsps/docs/loadingSummary.txt current loading summary]