| Version 4 (modified by , 14 years ago) ( diff ) |
|---|
Some notes about ippToPsps for the PSPS Operational Readiness Review (ORR)
Summary of loading to date
This sections provides a brief outline of loading to PSPS over the last two years. A more detailed summary can be found on the PSPS news page here.
2010
The first version of the ippToPsps code was complete by April 2010 after which we (slowly) loaded a total of 42,000 3PI batches for our beta testers. No stacks were available in DVO to be loaded to PSPS in 2010.
Loading was stopped and started throughout the year as we were testing the system, but it was clear that loading was too slow to keep up with the data flow promised from the IPP.
2011
More loading and testing early in the year.
Stacks were made available for the first time just prior to the Boston meeting in May. However, an unforeseen issue with the dense MD fields, it was impossible to extract data from DVO in time. This prompted us to create a DV) to MySQL system meaning a whole DVO database could be ingested in MySQL for faster processing through ippToPsps.
After the Boston meeting we started to load the new reprocessed data from the IPP (aka LAP). With the promise of a high throughput from IPP we needed to speed up ippToPsps, so a multi-client version was developed. Multiple instances of ippToPsps can be run on the same machine, or multiple machines, in order to load batches in parallel. By using a secure critical section clients cannot ....TODO
Another speed increase was obtained by implementing a stored-procedure in MySQL for the calculation of psf likelihoods. Ultimately, likelihoods will be provided by the IPP, but before that time it is the responsibility of ippToPsps.
During November and December we loaded half of all available old 3PI data from the IPP, while continuing to load new LAP data.
In december we also loaded new versions of the three-year SAS and MD04 surveys. Loading of MD04 would have been impossible without the ability to pre-load a MySQL database with all DVO data.
Summary of major speed improvements to ippToPsps
In summary, the major developments to improve loading speed in 2011 have been:
- multi-client version
- stored procedure to calculate likelihoods
- ability to pre-ingest dense regions of sky from DVO into MySQL for
Loading stress-test stats
With the code improvements described above, we stress-tested loading through ippToPsps during November and December 2011 by loading a large chunk of old 3PI data from the IPP. This loading of old data was performed between Oct 27th and Dec 1st, i.e. 5 weeks in which time we loaded all old data from the IPP from RA 6 to 18 hours, i.e. half the sky. This was roughly 55,000 frames. Some key points:
- this included the galactic center with somes frames containing up to 4 million detections
- this was done simultaneously with the loading of LAP frames and stacks
This stress-testing of ippToPsps showed that, for normal 3PI data in a quiet part of the sky, we can easily load ~100 frames per hour, or ~2400 exposures per day using multiple loading clients on up to 6 hosts. In short, ippToPsps can easily keep up with IPP production and, by using multiple clients, can quickly bulk-load whole surveys if required.
Anatomy of a batch
To give an idea of what takes time during loading with ippToPsps, this section gives a timing breakdown of a run-of-the-mill batch for a PS1 exposure with 83,542 detections. Below is a section of the ippToPsps log that detailed all stages of processing for a given batch.
2011-11-29 17:20:08 | INFO | 2011-11-29 17:20:08 | INFO | New P2 batch 2011-11-29 17:20:08 | INFO | 2011-11-29 17:20:08 | INFO | Batch name B00290162 2011-11-29 17:20:08 | INFO | Survey 3PI 2011-11-29 17:20:08 | INFO | Survey ID 0 2011-11-29 17:20:08 | INFO | Publishing to PSPS as survey OLD 2011-11-29 17:20:08 | INFO | DVO location /data/ipp045.0/eugene/3pi.20110819/catdir.20110819.v1 2011-11-29 17:20:08 | INFO | Use full DVO tables? no 2011-11-29 17:20:08 | INFO | Input FITS file /data/ipp040.0/nebulous/91/37/657669198.gpc1:ThreePi.nt:2011:01:29:o5590g0403o.289536:o5590g0403o.289536.cm.167125.smf 2011-11-29 17:20:08 | INFO | Input FITS primary header 66 cards found 2011-11-29 17:20:08 | INFO | Output path /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162 2011-11-29 17:20:08 | INFO | Cam ID 167125 2011-11-29 17:20:08 | INFO | Exp ID 289536 2011-11-29 17:20:08 | INFO | Exp name o5590g0403o 2011-11-29 17:20:08 | INFO | Distribution group ThreePi 2011-11-29 17:20:08 | INFO | Proccesing table FrameMeta 2011-11-29 17:20:08 | INFO | Reading FITS headers 2011-11-29 17:20:08 | INFO | Populating table ImageMeta 2011-11-29 17:20:10 | INFO | Running DVO ../src/dvograbber configs/oldthreepiGene2.xml /data/ipp045.0/eugene/3pi.20110819/catdir.20110819.v1 2011-11-29 17:23:07 | INFO | DVO access complete. Found 86494 detections 2011-11-29 17:23:07 | INFO | Importing tables with filter .*.psf 2011-11-29 17:23:20 | INFO | Done. Imported 60 tables 2011-11-29 17:23:20 | INFO | Creating indexes on IPP tables 2011-11-29 17:23:21 | INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+ 2011-11-29 17:23:21 | INFO | | OTA | Initial total | Sat Det | NULL instFlux | NULL peak ADU | NULL obj ID | Remainder | 2011-11-29 17:23:21 | INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+ 2011-11-29 17:23:21 | INFO | | XY01 | 1533 | 2 | 36 | 51 | 138 | 1306 | 2011-11-29 17:23:21 | INFO | | XY02 | 1606 | 0 | 21 | 40 | 90 | 1455 | 2011-11-29 17:23:21 | INFO | | XY03 | 1780 | 0 | 7 | 42 | 80 | 1651 | ... 2011-11-29 17:23:33 | INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+ 2011-11-29 17:23:33 | INFO | | Total | 97306 | 23 | 1160 | 2188 | 10393 | 83542 | 2011-11-29 17:23:33 | INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+ 2011-11-29 17:23:33 | INFO | Total detections 83542 2011-11-29 17:23:33 | INFO | Min objID 119151256944208155 2011-11-29 17:23:33 | INFO | Max objID 122921262940559465 2011-11-29 17:23:33 | INFO | Replacing NULLs with -999 2011-11-29 17:23:33 | INFO | Changing table names with regex ([a-zA-Z]+) 2011-11-29 17:23:36 | INFO | Writing to FITS /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162/00289536.FITS 2011-11-29 17:23:42 | INFO | Creating manifest /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162/BatchManifest.xml 2011-11-29 17:23:42 | INFO | Creating tar archive tar -cvf /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar -C /data/ipp005.0/rhenders/P2/ThreePi.V3 B00290162 2011-11-29 17:23:44 | INFO | Compressing tar archive gzip -c /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar > /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz 2011-11-29 17:23:46 | INFO | Attempting to publish /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz 2011-11-29 17:23:46 | INFO | Datastore publish successful
Breakdown
| Action | Time |
| reading the smf FITS file | 0:13 |
| DVO access | 2:57 |
| creating Db indexes, performing all numerical manipulations (calculating fluxes, likelihoods, removing duplicates, NULL fluxes etc etc) | 0:13 |
| creating FITS file, compressing and publishing to the datastore | 0:13 |
| Total | 3:38 |
Clearly DVO is the bottleneck. This example was a client running on the same machine as the DVO database (accessing over the network slows it down substantially). Also, during this batch creation, two other clients were running on the same machine using the same MySQL database.
because DVO stores data in one FITS per unit area on the sky, for densely populated regions, the resultant FITS files are huge and therefore extremely slow to access pre-Boston it wa taking 40 minutes oper frame. Ported all data to MySQL and access time dropped to 30 seconds.
Monitoring ippToPsps
Only on the IPP side of the interface do we have access to all information about all batches. From the IPP we know how many frames or stacks are available for a given survey, we know how many have processed through the interface and we know the progress of each item as it passed through the PSPS system
DVO processed through interface loaded to datastore loaded to ODM merge worthy merged
there are also various 'delete' stages: when a batch has become merge_worthy, it is safe to delete copies in the DXLayer and datastore. when a batch has merged it is safe to delete the copy of local IPP disk. all this needs to be monitored and so I put it all into Czartool, where we can see progress over time on time-series and rate plots.
