Changes between Version 1 and Version 2 of ippToPspsOrr

Timestamp:

Feb 15, 2012, 5:10:24 PM (14 years ago)

Author:

rhenders

Comment:

--

ippToPspsOrr

@@ -3 +3 @@
 
 [wiki:ippToPsps Back to main ippToPsps page]
+
+== Summary of loading to date ==
+
+This sections provides a brief outline of loading to PSPS over the last two years. A more detailed summary can be found on the PSPS news page [http://web01.psps.ifa.hawaii.edu/psps-trac/wiki/PSPSNews here].
+
+=== 2010 ===
+
+The first version of the ippToPsps code was complete by April 2010 after which we (slowly) loaded a total of 42,000 3PI batches for our beta testers. No stacks were available in DVO to be loaded to PSPS in 2010.
+
+Loading was stopped and started throughout the year as we were testing the system, but it was clear that loading was too slow to keep up with the data flow promised from the IPP.
+
+=== 2011 ===
+
+Stacks available for the first just prior to the Boston meeting in May.
+
+Many changes made to ippToPsps to improve speed in order to load nightly data as well as reprocessed:
+- multi-client: multiple instances of ippToPsps can be run on the same machine, or multiple machines
+- some processing (for likelihoods) now done by a MySQL stored procedure making a big speed increase
+- areas of sky with high coverage (eg MD fields) are incredibly slow to access from DVO, so for these area we can pre-ingest DVO data into MySQL to speed up loading.
+
+With these improvements we stress-tested loading during November and December 2011. We loaded the new LAP data as it became available from the IPP and loaded half the sky (6-18 hrs RA) of old 3PI data. This included the galactic center with somes frames containing up to 4 million detections.
+
+Between Oct 27th and Dec 1st, i.e. 5 weeks, we loaded all old data from the IPP from RA 6 to 18 hours, i.e. half the sky. This was 55,000.
+
+This was done simultaneously with the loading of LAP.
+
+In december we also loaded new versions of the three-year SAS and MD04 surveys. Loading of MD04 would have been impossible without the ability to pre-load a MySQL database with all DVO data.
+
+For normal 3PI data in a quiet part of the sky we can easily load ~100 per hour, ~2400 exposures per day using multiple loading clients over up to 6 hosts. This system is flexible and new clients can be started and stopped with ease depending on requirements.
+
+In short, ippToPsps can easily keep up with IPP production and, by using multiple clients, can quickly bulk-load whole surveys if required.
+
+== Anatomy of a batch ==
+
+This is an example of a run-of-the-mill batch for a PS1 exposure with 83,542 detections.
+
+
+ippToPsps needs to keep pace with IPP production of new and re-processed data. Initially (2010) loading was slow. A couple of bottlenecks were:
+- access to the DVO database is slow, especially for regions of the sky where we have high coverage (eg MD fields)
+- ippToPsps (hopefully temporarily) performs some processing, in particular a conversion of magnitudes to fluxes and the calculation of psf, extended and cosmic ray likelihoods.  Initially the likelihoods in particular were slow due to the use of an explicit database cursor, but I have since implemented a stored procedure in the MySQL database, which using an implicit cursor is much more efficient
+
+
+{{{
+
+
+2011-11-29 17:20:08 |    INFO | 
+2011-11-29 17:20:08 |    INFO |                             New P2 batch
+2011-11-29 17:20:08 |    INFO | 
+2011-11-29 17:20:08 |    INFO | Batch name                              B00290162
+2011-11-29 17:20:08 |    INFO | Survey                                  3PI
+2011-11-29 17:20:08 |    INFO | Survey ID                               0
+2011-11-29 17:20:08 |    INFO | Publishing to PSPS as survey            OLD
+2011-11-29 17:20:08 |    INFO | DVO location                            /data/ipp045.0/eugene/3pi.20110819/catdir.20110819.v1
+2011-11-29 17:20:08 |    INFO | Use full DVO tables?                    no
+2011-11-29 17:20:08 |    INFO | Input FITS file                         /data/ipp040.0/nebulous/91/37/657669198.gpc1:ThreePi.nt:2011:01:29:o5590g0403o.289536:o5590g0403o.289536.cm.167125.smf
+2011-11-29 17:20:08 |    INFO | Input FITS primary header               66 cards found
+2011-11-29 17:20:08 |    INFO | Output path                             /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162
+2011-11-29 17:20:08 |    INFO | Cam ID                                  167125
+2011-11-29 17:20:08 |    INFO | Exp ID                                  289536
+2011-11-29 17:20:08 |    INFO | Exp name                                o5590g0403o
+2011-11-29 17:20:08 |    INFO | Distribution group                      ThreePi
+2011-11-29 17:20:08 |    INFO | Proccesing table                        FrameMeta
+2011-11-29 17:20:08 |    INFO | Reading                                 FITS headers
+2011-11-29 17:20:08 |    INFO | Populating table                        ImageMeta
+2011-11-29 17:20:10 |    INFO | Running DVO                             ../src/dvograbber configs/oldthreepiGene2.xml /data/ipp045.0/eugene/3pi.20110819/catdir.20110819.v1
+2011-11-29 17:23:07 |    INFO | DVO access complete. Found              86494 detections
+2011-11-29 17:23:07 |    INFO | Importing tables with filter            .*.psf
+2011-11-29 17:23:20 |    INFO | Done. Imported                          60 tables
+2011-11-29 17:23:20 |    INFO | Creating indexes on                     IPP tables
+2011-11-29 17:23:21 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
+2011-11-29 17:23:21 |    INFO | |  OTA  | Initial total |   Sat Det     | NULL instFlux | NULL peak ADU | NULL obj ID   |  Remainder    |
+2011-11-29 17:23:21 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
+2011-11-29 17:23:21 |    INFO | |  XY01 |          1533 |             2 |            36 |            51 |           138 |          1306 |
+2011-11-29 17:23:21 |    INFO | |  XY02 |          1606 |             0 |            21 |            40 |            90 |          1455 |
+2011-11-29 17:23:21 |    INFO | |  XY03 |          1780 |             0 |             7 |            42 |            80 |          1651 |
+...
+2011-11-29 17:23:33 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
+2011-11-29 17:23:33 |    INFO | | Total |         97306 |            23 |          1160 |          2188 |         10393 |         83542 |
+2011-11-29 17:23:33 |    INFO | +-------+---------------+---------------+---------------+---------------+---------------+---------------+
+2011-11-29 17:23:33 |    INFO | Total detections                        83542
+2011-11-29 17:23:33 |    INFO | Min objID                               119151256944208155
+2011-11-29 17:23:33 |    INFO | Max objID                               122921262940559465
+2011-11-29 17:23:33 |    INFO | Replacing NULLs with                    -999
+2011-11-29 17:23:33 |    INFO | Changing table names with regex         ([a-zA-Z]+)
+2011-11-29 17:23:36 |    INFO | Writing to FITS                         /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162/00289536.FITS
+2011-11-29 17:23:42 |    INFO | Creating manifest                       /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162/BatchManifest.xml
+2011-11-29 17:23:42 |    INFO | Creating tar archive                    tar -cvf /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar -C /data/ipp005.0/rhenders/P2/ThreePi.V3 B00290162
+2011-11-29 17:23:44 |    INFO | Compressing tar archive                 gzip -c /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar > /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz
+2011-11-29 17:23:46 |    INFO | Attempting to publish                   /data/ipp005.0/rhenders/P2/ThreePi.V3/B00290162.tar.gz
+2011-11-29 17:23:46 |    INFO | Datastore publish                       successful
+
+}}}
+
+
+Breakdown:
+
+reading FITS file: 0:13
+DVO access: 2:57
+creating Db indexes, performing all numerical manipulations (calculating fluxes, likehoods, removing duplicates, NULL fluxes etc etc): 0:13
+creating FITS file, compressing and publishing to the datastore: 0:13
+
+TOTAL: 3:38
+
+Clearly DVO is the bottleneck. This example was a client running on the same machine as the DVO database. Accessing over the network slows it down substantially. Also, during this batch creation, two other clients were running on the same machine using the same MySQL database.
+
+
+because DVO stores data in one FITS per unit area on the sky, for densely populated regions, the resultant FITS files are huge and therefore extremely slow to access
+pre-Boston it wa taking 40 minutes oper frame. Ported all data to MySQL and access time dropped to 30 seconds.
+
+implemented a multi-client version of my interface code, meaning any number can be instantiated concurrently and, by using a secure critical section, they do not interfere with one another
+
+
+Monitoring
+
+Only on the IPP side of the interface do we have access to all information about all batches. From the IPP i know how many frames or stacks are available for a given survey, I know how many I have processed through my interface but I also know the progress of each item as it passed through the PSPS system:
+
+DVO
+processed through interface
+loaded to datastore
+loaded to ODM
+merge worthy
+merged
+
+there are also various 'delete' stages: when a batch has become merge_worthy, it is safe to delete copies in the DXLayer and datastore. when a batch has merged it is safe to delete the copy of local IPP disk. all this needs to be monitored and so I put it all into Czartool, where we can see progress over time on time-series and rate plots.
+
+