Changes between Version 4 and Version 5 of ippToPspsOrr
- Timestamp:
- Feb 15, 2012, 5:48:00 PM (14 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
ippToPspsOrr
v4 v5 10 10 This sections provides a brief outline of loading to PSPS over the last two years. A more detailed summary can be found on the PSPS news page [http://web01.psps.ifa.hawaii.edu/psps-trac/wiki/PSPSNews here]. 11 11 12 === 2010 === 12 April: first version of the ippToPsps code complete 13 April: started loading 3PI data for our beta testers 14 February: finished loading 42,000 3PI batches 13 15 14 The first version of the ippToPsps code was complete by April 2010 after which we (slowly) loaded a total of 42,000 3PI batches for our beta testers. No stacks were available in DVO to be loadedto PSPS in 2010.16 No stacks were made available to PSPS in 2010. 15 17 16 18 Loading was stopped and started throughout the year as we were testing the system, but it was clear that loading was too slow to keep up with the data flow promised from the IPP. 17 19 18 === 2011 ===19 20 20 21 More loading and testing early in the year. 21 22 22 Stacks were made available for the first time just prior to the Boston meeting in May. However, an unforeseen issue with the dense MD fields, it was impossible to extract data from DVO in time. This prompted us to create a DV) to MySQL system meaning a whole DVO database could be ingested in MySQL for faster processing through ippToPsps.23 Stacks were made available for the first time just prior to the Boston meeting in May. However, an unforeseen issue with the dense MD fields, it was impossible to extract data from DVO in time. 23 24 24 After the Boston meeting we started to load the new reprocessed data from the IPP (aka LAP). With the promise of a high throughput from IPP we needed to speed up ippToPsps, so a multi-client version was developed. Multiple instances of ippToPsps can be run on the same machine, or multiple machines, in order to load batches in parallel. By using a secure critical section clients cannot ....TODO 25 26 Another speed increase was obtained by implementing a stored-procedure in MySQL for the calculation of psf likelihoods. Ultimately, likelihoods will be provided by the IPP, but before that time it is the responsibility of ippToPsps. 25 After the Boston meeting we started to load the new reprocessed data from the IPP (aka LAP). 27 26 28 27 During November and December we loaded half of all available old 3PI data from the IPP, while continuing to load new LAP data. … … 30 29 In december we also loaded new versions of the three-year SAS and MD04 surveys. Loading of MD04 would have been impossible without the ability to pre-load a MySQL database with all DVO data. 31 30 32 == Summary of major speed improvements to ippToPsps==31 == Major speed improvements to ippToPsps in 2011 == 33 32 34 In summary, the major developments to improve loading speed in 2011 have been: 33 Certain major speed improvements were necessary in 2011. These are detailed below. 35 34 36 * multi-client version 37 * stored procedure to calculate likelihoods 38 * ability to pre-ingest dense regions of sky from DVO into MySQL for 35 === Pre-ingesting DVO into MySQL === 36 37 Because DVO stores data in FITS per unit area on the sky, for densely populated regions the resultant FITS files can be huge and therefore extremely slow to access. When given the DVO database for MD04 prior to the Bostom=n meeting, it was taking 40 minutes to access a single frame's worth of detections from DVO. 38 39 The original 40 minutes (above) dropped to 30 seconds, but of course ingest of the DVO data takes time. 40 41 === Multiple clients === 42 43 With the promise of a high throughput of new 'LAP' data from the IPP we needed to speed up ippToPsps, so a multi-client version was developed. Multiple instances of ippToPsps can be run on the same machine, or multiple machines, so that batches can be loaded in parallel. By using a secure critical section it is impossible for clients to attempt to load the same batch. 44 45 === Stored procedure to calculate likelihoods === 46 47 A certain amount of data processing is done within ippToPsps, hopefully temporarily. One particularly time-intensive example of this the calculation of psf likelihoods. Ultimately, likelihoods will be provided by the IPP, but before that time it is the responsibility of ippToPsps and to speed this up a stored procedure was implemented within MySQL. 39 48 40 49 == Loading stress-test stats == … … 113 122 114 123 115 because DVO stores data in one FITS per unit area on the sky, for densely populated regions, the resultant FITS files are huge and therefore extremely slow to access116 pre-Boston it wa taking 40 minutes oper frame. Ported all data to MySQL and access time dropped to 30 seconds.117 124 118 125 … … 130 137 131 138 there are also various 'delete' stages: when a batch has become merge_worthy, it is safe to delete copies in the DXLayer and datastore. when a batch has merged it is safe to delete the copy of local IPP disk. all this needs to be monitored and so I put it all into Czartool, where we can see progress over time on time-series and rate plots. 132
