Changes between Version 5 and Version 6 of ippToPspsOrr
- Timestamp:
- Feb 16, 2012, 3:58:46 PM (14 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
ippToPspsOrr
v5 v6 8 8 == Summary of loading to date == 9 9 10 This sections provides a brief o utline of loading to PSPSover the last two years. A more detailed summary can be found on the PSPS news page [http://web01.psps.ifa.hawaii.edu/psps-trac/wiki/PSPSNews here].10 This sections provides a brief overview of loading activity over the last two years. A more detailed summary can be found on the PSPS news page [http://web01.psps.ifa.hawaii.edu/psps-trac/wiki/PSPSNews here]. 11 11 12 April: first version of the ippToPsps code complete 13 April: started loading 3PI data for our beta testers 14 February: finished loading 42,000 3PI batches12 * 2010 13 * '''April''': first version of the ippToPsps code complete 14 * '''April''': started loading 3PI data for our beta testers 15 15 16 No stacks were made available to PSPS in 2010. 16 * 2011: 17 * '''February''': finished loading 42,000 3PI batches for beta testers 18 * '''May''': stacks first made available to PSPS 19 * '''May''': MD4, including stacks, loaded for PS1SC Boston meeting 20 * '''July''': deleted beta tester's data 21 * '''August''': started loading re-processed (aka LAP) data from the IPP 22 * '''October''': started loading old 3PI data from IPP 23 * '''December''': finished loading half the sky of old 3PI (6 - 18 hrs RA) 24 * '''December''': loaded new MD04 and SAS 3-year surveys in readiness for PS1SC meeting 25 * '''December''': stopped all loading due to merge problems withing PSPS 17 26 18 Loading was stopped and started throughout the year as we were testing the system, but it was clear that loading was too slow to keep up with the data flow promised from the IPP. 27 * 2012: 28 * '''February''': re-started loading of re-processed 3PI survey 19 29 20 21 More loading and testing early in the year.22 23 Stacks were made available for the first time just prior to the Boston meeting in May. However, an unforeseen issue with the dense MD fields, it was impossible to extract data from DVO in time.24 25 After the Boston meeting we started to load the new reprocessed data from the IPP (aka LAP).26 27 During November and December we loaded half of all available old 3PI data from the IPP, while continuing to load new LAP data.28 29 In december we also loaded new versions of the three-year SAS and MD04 surveys. Loading of MD04 would have been impossible without the ability to pre-load a MySQL database with all DVO data.30 30 31 31 == Major speed improvements to ippToPsps in 2011 == … … 35 35 === Pre-ingesting DVO into MySQL === 36 36 37 Because DVO stores data in FITS per unit area on the sky, for densely populated regions the resultant FITS files can be huge and therefore extremely slow to access. When given the DVO database for MD04 prior to the Bostom=n meeting, it was taking 40 minutes to access a single frame's worth of detections from DVO. 38 39 The original 40 minutes (above) dropped to 30 seconds, but of course ingest of the DVO data takes time. 37 When a DVO database was provided to PSPS for MD04 prior to the Boston meeting we encountered an unforeseen speed issue. DVO stores data in FITS files, each one representing an area on the sky. For regions where we have high coverage, eg medium deep fields, the FITS files are orders of magnitude larger than those for the 3PI survey. This means that they are extremely slow to access. It was taking about 40 minutes to access one frame of data from the MD04 DVO database provided prior to the Boston meeting. By writing code that ingested this entire DVO database into a MySQL database this 40 minute access time dropped to 30 seconds. The full ingest took approximately 24 hours. 40 38 41 39 === Multiple clients === 42 40 43 With the promise of a high throughput of new 'LAP' data from the IPP we needed to speed up ippToPsps , so a multi-client version was developed. Multiple instances of ippToPsps can be run on the same machine, or multiple machines, so that batches can be loaded in parallel. By using a secure critical section it is impossible forclients to attempt to load the same batch.41 With the promise of a high throughput of new 'LAP' data from the IPP we needed to speed up ippToPsps so a multi-client version was developed. Multiple instances of ippToPsps can be run on the same machine, or multiple machines, so that batches can be loaded in parallel. The use of a secure critical section makes it impossible for different clients to attempt to load the same batch. 44 42 45 43 === Stored procedure to calculate likelihoods === 46 44 47 A certain amount of data processing is done within ippToPsps , hopefully temporarily. One particularly time-intensive example of this the calculation of psf likelihoods. Ultimately, likelihoods will be provided by the IPP, but before that time it is the responsibility of ippToPsps and to speed this up a stored procedure was implemented within MySQL.45 A certain amount of data processing is done within ippToPsps (hopefully temporarily). One particularly time-intensive example of this the calculation of psf likelihoods. Ultimately, likelihoods will be provided by the IPP, but before that time it is the responsibility of ippToPsps and to speed this up a stored procedure was implemented within MySQL. 48 46 49 47 == Loading stress-test stats == … … 52 50 53 51 * this included the galactic center with somes frames containing up to 4 million detections 54 * this was done simultaneously with the loading of LAP frames and stacks 52 * this was done simultaneously with the loading of LAP frames and stacks as well as early versions of MD4 and SA3 55 53 56 54 This stress-testing of ippToPsps showed that, for normal 3PI data in a quiet part of the sky, we can easily load '''~100 frames per hour''', or '''~2400 exposures per day''' using multiple loading clients on up to 6 hosts. In short, ippToPsps can easily keep up with IPP production and, by using multiple clients, can quickly bulk-load whole surveys if required. 57 55 58 59 56 == Anatomy of a batch == 60 57 61 To give an idea of what takes time during loading with ippToPsps, this section gives a timing breakdown of a run-of-the-mill batch for a PS1 exposure with 83,542 detections. Below is a section of the ippToPsps log that detailed all stages of processing for a givenbatch.58 To give an idea of what takes time during loading with ippToPsps, this section gives a timing breakdown of a run-of-the-mill batch for a PS1 exposure with 83,542 detections. Below is a section of the ippToPsps log detailing all stages of processing for this batch. 62 59 63 60 {{{ … … 119 116 || '''Total''' || '''3:38''' || 120 117 121 Clearly DVO is the bottleneck. This example was a client running on the same machine as the DVO database (accessing over the network slows it down substantially). Also, during this batch creation, two other clients were running on the same machine using the same MySQL database. 122 123 124 125 118 Clearly DVO is the bottleneck. This example is from a client running on the same machine as the DVO database (accessing over the network slows down DVO access substantially). Also, during this batch creation, two other clients were running on the same machine using the same MySQL database. 126 119 127 120 == Monitoring ippToPsps == 128 121 129 Only on the IPP side of the interface do we have access to all information about all batches. From the IPP we know how many frames or stacks are available for a given survey, we know how many have processed through the interface and we know the progress of each item as it passed through the PSPS system 130 131 DVO 132 processed through interface 133 loaded to datastore 134 loaded to ODM 135 merge worthy 136 merged 137 138 there are also various 'delete' stages: when a batch has become merge_worthy, it is safe to delete copies in the DXLayer and datastore. when a batch has merged it is safe to delete the copy of local IPP disk. all this needs to be monitored and so I put it all into Czartool, where we can see progress over time on time-series and rate plots. 122 Only on the IPP side of the interface do we have access to all information about a given batches. From the IPP we know how many frames or stacks are available for a given survey, we know how many have processed through the interface and we know the progress of each item as it passed through the PSPS system. This can all be monitored through a special 'czartool' page on ippMonitor, complete with time-series and rate plots.
