| Version 5 (modified by , 14 years ago) ( diff ) |
|---|
Some notes about ippToPsps for the PSPS Operational Readiness Review (ORR)
Summary of loading to date
This sections provides a brief outline of loading to PSPS over the last two years. A more detailed summary can be found on the PSPS news page here.
April: first version of the ippToPsps code complete April: started loading 3PI data for our beta testers February: finished loading 42,000 3PI batches
No stacks were made available to PSPS in 2010.
Loading was stopped and started throughout the year as we were testing the system, but it was clear that loading was too slow to keep up with the data flow promised from the IPP.
More loading and testing early in the year.
Stacks were made available for the first time just prior to the Boston meeting in May. However, an unforeseen issue with the dense MD fields, it was impossible to extract data from DVO in time.
After the Boston meeting we started to load the new reprocessed data from the IPP (aka LAP).
During November and December we loaded half of all available old 3PI data from the IPP, while continuing to load new LAP data.
In december we also loaded new versions of the three-year SAS and MD04 surveys. Loading of MD04 would have been impossible without the ability to pre-load a MySQL database with all DVO data.
Major speed improvements to ippToPsps in 2011
Certain major speed improvements were necessary in 2011. These are detailed below.
Pre-ingesting DVO into MySQL
Because DVO stores data in FITS per unit area on the sky, for densely populated regions the resultant FITS files can be huge and therefore extremely slow to access. When given the DVO database for MD04 prior to the Bostom=n meeting, it was taking 40 minutes to access a single frame's worth of detections from DVO.
The original 40 minutes (above) dropped to 30 seconds, but of course ingest of the DVO data takes time.
Multiple clients
With the promise of a high throughput of new 'LAP' data from the IPP we needed to speed up ippToPsps, so a multi-client version was developed. Multiple instances of ippToPsps can be run on the same machine, or multiple machines, so that batches can be loaded in parallel. By using a secure critical section it is impossible for clients to attempt to load the same batch.
Stored procedure to calculate likelihoods
A certain amount of data processing is done within ippToPsps, hopefully temporarily. One particularly time-intensive example of this the calculation of psf likelihoods. Ultimately, likelihoods will be provided by the IPP, but before that time it is the responsibility of ippToPsps and to speed this up a stored procedure was implemented within MySQL.
Loading stress-test stats
With the code improvements described above, we stress-tested loading through ippToPsps during November and December 2011 by loading a large chunk of old 3PI data from the IPP. This loading of old data was performed between Oct 27th and Dec 1st, i.e. 5 weeks in which time we loaded all old data from the IPP from RA 6 to 18 hours, i.e. half the sky. This was roughly 55,000 frames. Some key points:
- this included the galactic center with somes frames containing up to 4 million detections
- this was done simultaneously with the loading of LAP frames and stacks
This stress-testing of ippToPsps showed that, for normal 3PI data in a quiet part of the sky, we can easily load ~100 frames per hour, or ~2400 exposures per day using multiple loading clients on up to 6 hosts. In short, ippToPsps can easily keep up with IPP production and, by using multiple clients, can quickly bulk-load whole surveys if required.
Anatomy of a batch
To give an idea of what takes time during loading with ippToPsps, this section gives a timing breakdown of a run-of-the-mill batch for a PS1 exposure with 83,542 detections. Below is a section of the ippToPsps log that detailed all stages of processing for a given batch.
2011-11-29 17:20:08 | INFO | 2011-11-29 17:20:08 | INFO | New P2 batch 2011-11-29 17:20:08 | INFO | 2011-11-29 17:20:08 | INFO | Batch name B00290162 2011-11-29 17:20:08 | INFO | Survey 3PI 2011-11-29 17:20:08 | INFO | Survey ID 0 2011-11-29 17:20:08 | INFO | Publishing to PSPS as survey OLD 2011-11-29 17:20:08 | INFO | DVO location /data/ipp045.0/eugene/3pi.20110819/catdir.20110819.v1 2011-11-29 17:20:08 | INFO | Use full DVO tables? no 2011-11-29 17:20:08 | INFO | Input FITS file /data/ipp040.0/nebulous/91/37/657669198.gpc1:ThreePi.nt:2011:01:29:o5590g0403o.289536:o5590g0403o.289536.cm.167125.smf 2011-11-29 17:20:08 | INFO | Input FITS primary header 66 cards found 2011-11-29 17:20:08 | INFO | Output path /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162 2011-11-29 17:20:08 | INFO | Cam ID 167125 2011-11-29 17:20:08 | INFO | Exp ID 289536 2011-11-29 17:20:08 | INFO | Exp name o5590g0403o 2011-11-29 17:20:08 | INFO | Distribution group ThreePi 2011-11-29 17:20:08 | INFO | Proccesing table FrameMeta 2011-11-29 17:20:08 | INFO | Reading FITS headers 2011-11-29 17:20:08 | INFO | Populating table ImageMeta 2011-11-29 17:20:10 | INFO | Running DVO ../src/dvograbber configs/oldthreepiGene2.xml /data/ipp045.0/eugene/3pi.20110819/catdir.20110819.v1 2011-11-29 17:23:07 | INFO | DVO access complete. Found 86494 detections 2011-11-29 17:23:07 | INFO | Importing tables with filter .*.psf 2011-11-29 17:23:20 | INFO | Done. Imported 60 tables 2011-11-29 17:23:20 | INFO | Creating indexes on IPP tables 2011-11-29 17:23:21 | INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+ 2011-11-29 17:23:21 | INFO | | OTA | Initial total | Sat Det | NULL instFlux | NULL peak ADU | NULL obj ID | Remainder | 2011-11-29 17:23:21 | INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+ 2011-11-29 17:23:21 | INFO | | XY01 | 1533 | 2 | 36 | 51 | 138 | 1306 | 2011-11-29 17:23:21 | INFO | | XY02 | 1606 | 0 | 21 | 40 | 90 | 1455 | 2011-11-29 17:23:21 | INFO | | XY03 | 1780 | 0 | 7 | 42 | 80 | 1651 | ... 2011-11-29 17:23:33 | INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+ 2011-11-29 17:23:33 | INFO | | Total | 97306 | 23 | 1160 | 2188 | 10393 | 83542 | 2011-11-29 17:23:33 | INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+ 2011-11-29 17:23:33 | INFO | Total detections 83542 2011-11-29 17:23:33 | INFO | Min objID 119151256944208155 2011-11-29 17:23:33 | INFO | Max objID 122921262940559465 2011-11-29 17:23:33 | INFO | Replacing NULLs with -999 2011-11-29 17:23:33 | INFO | Changing table names with regex ([a-zA-Z]+) 2011-11-29 17:23:36 | INFO | Writing to FITS /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162/00289536.FITS 2011-11-29 17:23:42 | INFO | Creating manifest /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162/BatchManifest.xml 2011-11-29 17:23:42 | INFO | Creating tar archive tar -cvf /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar -C /data/ipp005.0/rhenders/P2/ThreePi.V3 B00290162 2011-11-29 17:23:44 | INFO | Compressing tar archive gzip -c /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar > /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz 2011-11-29 17:23:46 | INFO | Attempting to publish /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz 2011-11-29 17:23:46 | INFO | Datastore publish successful
Breakdown
| Action | Time |
| reading the smf FITS file | 0:13 |
| DVO access | 2:57 |
| creating Db indexes, performing all numerical manipulations (calculating fluxes, likelihoods, removing duplicates, NULL fluxes etc etc) | 0:13 |
| creating FITS file, compressing and publishing to the datastore | 0:13 |
| Total | 3:38 |
Clearly DVO is the bottleneck. This example was a client running on the same machine as the DVO database (accessing over the network slows it down substantially). Also, during this batch creation, two other clients were running on the same machine using the same MySQL database.
Monitoring ippToPsps
Only on the IPP side of the interface do we have access to all information about all batches. From the IPP we know how many frames or stacks are available for a given survey, we know how many have processed through the interface and we know the progress of each item as it passed through the PSPS system
DVO processed through interface loaded to datastore loaded to ODM merge worthy merged
there are also various 'delete' stages: when a batch has become merge_worthy, it is safe to delete copies in the DXLayer and datastore. when a batch has merged it is safe to delete the copy of local IPP disk. all this needs to be monitored and so I put it all into Czartool, where we can see progress over time on time-series and rate plots.
