Some notes about ippToPsps for the PSPS Operational Readiness Review (ORR)

Back to main ippToPsps page

Summary of loading to date

This sections provides a brief outline of loading to PSPS over the last two years. A more detailed summary can be found on the PSPS news page ​here.

2010

The first version of the ippToPsps code was complete by April 2010 after which we (slowly) loaded a total of 42,000 3PI batches for our beta testers. No stacks were available in DVO to be loaded to PSPS in 2010.

Loading was stopped and started throughout the year as we were testing the system, but it was clear that loading was too slow to keep up with the data flow promised from the IPP.

2011

Stacks available for the first just prior to the Boston meeting in May.

Many changes made to ippToPsps to improve speed in order to load nightly data as well as reprocessed:

• multi-client: multiple instances of ippToPsps can be run on the same machine, or multiple machines
• some processing (for likelihoods) now done by a MySQL stored procedure making a big speed increase
• areas of sky with high coverage (eg MD fields) are incredibly slow to access from DVO, so for these area we can pre-ingest DVO data into MySQL to speed up loading.

With these improvements we stress-tested loading during November and December 2011. We loaded the new LAP data as it became available from the IPP and loaded half the sky (6-18 hrs RA) of old 3PI data. This included the galactic center with somes frames containing up to 4 million detections.

Between Oct 27th and Dec 1st, i.e. 5 weeks, we loaded all old data from the IPP from RA 6 to 18 hours, i.e. half the sky. This was 55,000.

This was done simultaneously with the loading of LAP.

In december we also loaded new versions of the three-year SAS and MD04 surveys. Loading of MD04 would have been impossible without the ability to pre-load a MySQL database with all DVO data.

For normal 3PI data in a quiet part of the sky we can easily load ~100 per hour, ~2400 exposures per day using multiple loading clients over up to 6 hosts. This system is flexible and new clients can be started and stopped with ease depending on requirements.

In short, ippToPsps can easily keep up with IPP production and, by using multiple clients, can quickly bulk-load whole surveys if required.

Anatomy of a batch

This is an example of a run-of-the-mill batch for a PS1 exposure with 83,542 detections.

ippToPsps needs to keep pace with IPP production of new and re-processed data. Initially (2010) loading was slow. A couple of bottlenecks were:

• access to the DVO database is slow, especially for regions of the sky where we have high coverage (eg MD fields)
• ippToPsps (hopefully temporarily) performs some processing, in particular a conversion of magnitudes to fluxes and the calculation of psf, extended and cosmic ray likelihoods. Initially the likelihoods in particular were slow due to the use of an explicit database cursor, but I have since implemented a stored procedure in the MySQL database, which using an implicit cursor is much more efficient

2011-11-29 17:20:08 |    INFO | 
2011-11-29 17:20:08 |    INFO |                             New P2 batch
2011-11-29 17:20:08 |    INFO | 
2011-11-29 17:20:08 |    INFO | Batch name                              B00290162
2011-11-29 17:20:08 |    INFO | Survey                                  3PI
2011-11-29 17:20:08 |    INFO | Survey ID                               0
2011-11-29 17:20:08 |    INFO | Publishing to PSPS as survey            OLD
2011-11-29 17:20:08 |    INFO | DVO location                            /data/ipp045.0/eugene/3pi.20110819/catdir.20110819.v1
2011-11-29 17:20:08 |    INFO | Use full DVO tables?                    no
2011-11-29 17:20:08 |    INFO | Input FITS file                         /data/ipp040.0/nebulous/91/37/657669198.gpc1:ThreePi.nt:2011:01:29:o5590g0403o.289536:o5590g0403o.289536.cm.167125.smf
2011-11-29 17:20:08 |    INFO | Input FITS primary header               66 cards found
2011-11-29 17:20:08 |    INFO | Output path                             /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162
2011-11-29 17:20:08 |    INFO | Cam ID                                  167125
2011-11-29 17:20:08 |    INFO | Exp ID                                  289536
2011-11-29 17:20:08 |    INFO | Exp name                                o5590g0403o
2011-11-29 17:20:08 |    INFO | Distribution group                      ThreePi
2011-11-29 17:20:08 |    INFO | Proccesing table                        FrameMeta
2011-11-29 17:20:08 |    INFO | Reading                                 FITS headers
2011-11-29 17:20:08 |    INFO | Populating table                        ImageMeta
2011-11-29 17:20:10 |    INFO | Running DVO                             ../src/dvograbber configs/oldthreepiGene2.xml /data/ipp045.0/eugene/3pi.20110819/catdir.20110819.v1
2011-11-29 17:23:07 |    INFO | DVO access complete. Found              86494 detections
2011-11-29 17:23:07 |    INFO | Importing tables with filter            .*.psf
2011-11-29 17:23:20 |    INFO | Done. Imported                          60 tables
2011-11-29 17:23:20 |    INFO | Creating indexes on                     IPP tables
2011-11-29 17:23:21 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
2011-11-29 17:23:21 |    INFO | |  OTA  | Initial total |   Sat Det     | NULL instFlux | NULL peak ADU | NULL obj ID   |  Remainder    |
2011-11-29 17:23:21 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
2011-11-29 17:23:21 |    INFO | |  XY01 |          1533 |             2 |            36 |            51 |           138 |          1306 |
2011-11-29 17:23:21 |    INFO | |  XY02 |          1606 |             0 |            21 |            40 |            90 |          1455 |
2011-11-29 17:23:21 |    INFO | |  XY03 |          1780 |             0 |             7 |            42 |            80 |          1651 |
...
2011-11-29 17:23:33 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
2011-11-29 17:23:33 |    INFO | | Total |         97306 |            23 |          1160 |          2188 |         10393 |         83542 |
2011-11-29 17:23:33 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
2011-11-29 17:23:33 |    INFO | Total detections                        83542
2011-11-29 17:23:33 |    INFO | Min objID                               119151256944208155
2011-11-29 17:23:33 |    INFO | Max objID                               122921262940559465
2011-11-29 17:23:33 |    INFO | Replacing NULLs with                    -999
2011-11-29 17:23:33 |    INFO | Changing table names with regex         ([a-zA-Z]+)
2011-11-29 17:23:36 |    INFO | Writing to FITS                         /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162/00289536.FITS
2011-11-29 17:23:42 |    INFO | Creating manifest                       /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162/BatchManifest.xml
2011-11-29 17:23:42 |    INFO | Creating tar archive                    tar -cvf /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar -C /data/ipp005.0/rhenders/P2/ThreePi.V3 B00290162
2011-11-29 17:23:44 |    INFO | Compressing tar archive                 gzip -c /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar > /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz
2011-11-29 17:23:46 |    INFO | Attempting to publish                   /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz
2011-11-29 17:23:46 |    INFO | Datastore publish                       successful

Breakdown:

reading FITS file: 0:13 DVO access: 2:57 creating Db indexes, performing all numerical manipulations (calculating fluxes, likehoods, removing duplicates, NULL fluxes etc etc): 0:13 creating FITS file, compressing and publishing to the datastore: 0:13

TOTAL: 3:38

Clearly DVO is the bottleneck. This example was a client running on the same machine as the DVO database. Accessing over the network slows it down substantially. Also, during this batch creation, two other clients were running on the same machine using the same MySQL database.

because DVO stores data in one FITS per unit area on the sky, for densely populated regions, the resultant FITS files are huge and therefore extremely slow to access pre-Boston it wa taking 40 minutes oper frame. Ported all data to MySQL and access time dropped to 30 seconds.

implemented a multi-client version of my interface code, meaning any number can be instantiated concurrently and, by using a secure critical section, they do not interfere with one another

Monitoring

Only on the IPP side of the interface do we have access to all information about all batches. From the IPP i know how many frames or stacks are available for a given survey, I know how many I have processed through my interface but I also know the progress of each item as it passed through the PSPS system:

DVO processed through interface loaded to datastore loaded to ODM merge worthy merged

there are also various 'delete' stages: when a batch has become merge_worthy, it is safe to delete copies in the DXLayer and datastore. when a batch has merged it is safe to delete the copy of local IPP disk. all this needs to be monitored and so I put it all into Czartool, where we can see progress over time on time-series and rate plots.