Some notes about ippToPsps for the PSPS Operational Readiness Review (ORR)

Back to main ippToPsps page

Summary of loading to date

This sections provides a brief overview of loading activity over the last two years. A more detailed summary can be found on the PSPS news page ​here.

• 2010
  • April: first version of the ippToPsps code complete
  • April: started loading 3PI data for our beta testers

• 2011:
  • February: finished loading 42,000 3PI batches for beta testers
  • May: stacks first made available to PSPS
  • May: MD4, including stacks, loaded for PS1SC Boston meeting
  • July: deleted beta tester's data
  • August: started loading re-processed (aka LAP) data from the IPP
  • October: started loading old 3PI data from IPP
  • December: finished loading half the sky of old 3PI (6 - 18 hrs RA)
  • December: loaded new MD04 and SAS 3-year surveys in readiness for PS1SC meeting
  • December: stopped all loading due to merge problems withing PSPS

• 2012:
  • February: re-started loading of re-processed 3PI survey

Major speed improvements to ippToPsps in 2011

Certain major speed improvements were necessary in 2011. These are detailed below.

Pre-ingesting DVO into MySQL

When a DVO database was provided to PSPS for MD04 prior to the Boston meeting we encountered an unforeseen speed issue. DVO stores data in FITS files, each one representing an area on the sky. For regions where we have high coverage, eg medium deep fields, the FITS files can be orders-of-magnitude larger than those for the 3PI survey. This means that they are extremely slow to access. For the MD04 DVO it was taking about 40 minutes to access one frame of data. By writing code that ingested this entire DVO database into a MySQL database this 40 minute access time dropped to 30 seconds. The full ingest took approximately 24 hours.

Multiple clients

With the promise of a high throughput of new 'LAP' data from the IPP we needed to speed up loading, so a multi-client version of ippToPsps was developed. Multiple instances of ippToPsps can be run on the same machine, or multiple machines, so that batches can be loaded in parallel. The use of a secure critical section makes it impossible for different clients to attempt to load the same batch.

Stored procedure to calculate likelihoods

A certain amount of data processing is done within ippToPsps (hopefully temporarily). One particularly time-intensive example of this the calculation of psf likelihoods. Ultimately, likelihoods will be provided by the IPP, but before that time it is the responsibility of ippToPsps and to speed this up a stored procedure was implemented within MySQL.

Loading stress-test stats

With the code improvements described above, we stress-tested loading through ippToPsps during November and December 2011 by loading a large chunk of old 3PI data from the IPP. This loading of old data was performed between Oct 27th and Dec 1st, i.e. 5 weeks in which time we loaded all old data from the IPP from RA 6 to 18 hours, i.e. half the sky. This was roughly 55,000 frames. Some key points:

• this included the galactic center with somes frames containing up to 4 million detections
• this was done simultaneously with the loading of LAP frames and stacks as well as early versions of MD4 and SA3

This stress-testing of ippToPsps showed that, for normal 3PI data in a quiet part of the sky, we can easily load ~100 frames per hour, or ~2400 exposures per day using multiple loading clients on up to 6 hosts. In short, ippToPsps can easily keep up with IPP production and, by using multiple clients, can quickly bulk-load whole surveys if required.

Anatomy of a batch

To give an idea of what takes time during loading with ippToPsps, this section gives a timing breakdown of a run-of-the-mill batch for a PS1 exposure with 83,542 detections. Below is a section of the ippToPsps log detailing all stages of processing for this batch.

2011-11-29 17:20:08 |    INFO | 
2011-11-29 17:20:08 |    INFO |                             New P2 batch
2011-11-29 17:20:08 |    INFO | 
2011-11-29 17:20:08 |    INFO | Batch name                              B00290162
2011-11-29 17:20:08 |    INFO | Survey                                  3PI
2011-11-29 17:20:08 |    INFO | Survey ID                               0
2011-11-29 17:20:08 |    INFO | Publishing to PSPS as survey            OLD
2011-11-29 17:20:08 |    INFO | DVO location                            /data/ipp045.0/eugene/3pi.20110819/catdir.20110819.v1
2011-11-29 17:20:08 |    INFO | Use full DVO tables?                    no
2011-11-29 17:20:08 |    INFO | Input FITS file                         /data/ipp040.0/nebulous/91/37/657669198.gpc1:ThreePi.nt:2011:01:29:o5590g0403o.289536:o5590g0403o.289536.cm.167125.smf
2011-11-29 17:20:08 |    INFO | Input FITS primary header               66 cards found
2011-11-29 17:20:08 |    INFO | Output path                             /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162
2011-11-29 17:20:08 |    INFO | Cam ID                                  167125
2011-11-29 17:20:08 |    INFO | Exp ID                                  289536
2011-11-29 17:20:08 |    INFO | Exp name                                o5590g0403o
2011-11-29 17:20:08 |    INFO | Distribution group                      ThreePi
2011-11-29 17:20:08 |    INFO | Proccesing table                        FrameMeta
2011-11-29 17:20:08 |    INFO | Reading                                 FITS headers
2011-11-29 17:20:08 |    INFO | Populating table                        ImageMeta
2011-11-29 17:20:10 |    INFO | Running DVO                             ../src/dvograbber configs/oldthreepiGene2.xml /data/ipp045.0/eugene/3pi.20110819/catdir.20110819.v1
2011-11-29 17:23:07 |    INFO | DVO access complete. Found              86494 detections
2011-11-29 17:23:07 |    INFO | Importing tables with filter            .*.psf
2011-11-29 17:23:20 |    INFO | Done. Imported                          60 tables
2011-11-29 17:23:20 |    INFO | Creating indexes on                     IPP tables
2011-11-29 17:23:21 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
2011-11-29 17:23:21 |    INFO | |  OTA  | Initial total |   Sat Det     | NULL instFlux | NULL peak ADU | NULL obj ID   |  Remainder    |
2011-11-29 17:23:21 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
2011-11-29 17:23:21 |    INFO | |  XY01 |          1533 |             2 |            36 |            51 |           138 |          1306 |
2011-11-29 17:23:21 |    INFO | |  XY02 |          1606 |             0 |            21 |            40 |            90 |          1455 |
2011-11-29 17:23:21 |    INFO | |  XY03 |          1780 |             0 |             7 |            42 |            80 |          1651 |
...
2011-11-29 17:23:33 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
2011-11-29 17:23:33 |    INFO | | Total |         97306 |            23 |          1160 |          2188 |         10393 |         83542 |
2011-11-29 17:23:33 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
2011-11-29 17:23:33 |    INFO | Total detections                        83542
2011-11-29 17:23:33 |    INFO | Min objID                               119151256944208155
2011-11-29 17:23:33 |    INFO | Max objID                               122921262940559465
2011-11-29 17:23:33 |    INFO | Replacing NULLs with                    -999
2011-11-29 17:23:33 |    INFO | Changing table names with regex         ([a-zA-Z]+)
2011-11-29 17:23:36 |    INFO | Writing to FITS                         /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162/00289536.FITS
2011-11-29 17:23:42 |    INFO | Creating manifest                       /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162/BatchManifest.xml
2011-11-29 17:23:42 |    INFO | Creating tar archive                    tar -cvf /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar -C /data/ipp005.0/rhenders/P2/ThreePi.V3 B00290162
2011-11-29 17:23:44 |    INFO | Compressing tar archive                 gzip -c /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar > /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz
2011-11-29 17:23:46 |    INFO | Attempting to publish                   /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz
2011-11-29 17:23:46 |    INFO | Datastore publish                       successful

Breakdown

Action	Time
reading the smf FITS file	0:13
DVO access	2:57
creating Db indexes, performing all numerical manipulations (calculating fluxes, likelihoods, removing duplicates, NULL fluxes etc etc)	0:13
creating FITS file, compressing and publishing to the datastore	0:13
Total	3:38

Clearly DVO is the bottleneck. This example is from a client running on the same machine as the DVO database (accessing over the network slows down DVO access substantially). Also, during this batch creation, two other clients were running on the same machine using the same MySQL database.

Monitoring ippToPsps

Only on the IPP side of the interface do we have access to all information about a given batche. From the IPP we know how many frames or stacks are available for a given survey, we know how many have processed through the interface and we know the progress of each item as it passed through the PSPS system. This can all be monitored through a special 'czartool' page on ippMonitor, complete with time-series and rate plots.