Changes between Version 6 and Version 7 of IppToPspsDatastorePV3Descriptions
- Timestamp:
- Nov 5, 2015, 10:53:16 PM (11 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
IppToPspsDatastorePV3Descriptions
v6 v7 1 There is a desire to organize the datastores for PV3 into sky chunks that are of a similar size of data, organized by dxlayer and load merge machine. The intent is to always feed the PSPS a chunk of batches that add up to a similar size in bytes for each PSPS machine so there are no surprises. Over the course of the ingest and merge, the only thing that will significantly vary as we ingest across the sky is2 the cumulative size of the merged database. Each subsequent addition prior to merge, will be about the same size, and will be handled by a similar machine. Our greatest uncertainty at this stage is the scaling with the total size, not the scaling per chunk. This page describes Ken's suggested datastore tessellation, as 3 modified by Heather. 1 There is a desire to organize the datastores for PV3 into sky chunks that are of a similar size of data, organized by dxlayer and load merge machine. The intent is to always feed the PSPS a chunk of batches that add up to a similar size in bytes for each PSPS machine so there are no surprises. It also follows the new PSPS hardware design including the switches and network. 2 See [[https://ps1wiki.ifa.hawaii.edu/trac/wiki/PSPSClusterHardwareEstimate2015 | PSPS Hardware Plan C] Over the course of the ingest and merge, the only thing that will significantly vary as we ingest across the sky is 3 the cumulative size of the merged database. Each subsequent addition prior to merge, will be about the same size, and will be handled by a similar machine. Our greatest uncertainty at this stage is the scaling with the total size, not the scaling per chunk. This page describes a "datastore tessellation". 4 4 5 5 == Step 1 == 6 6 7 Subdivide sky into the 32 declination layers. Each of the 32 layers has an equal amount of data in each layer (so, the thickness of the layers will vary, in order to increase/decrease the amount of data. These layers correspond to one each of the new PSPS Load/Merge machines. 7 Subdivide sky into the 32 declination layers. Each of the 32 layers has an equal amount of data in each layer (so, the thickness of the layers will vary, in order to increase/decrease the amount of data. These layers correspond to one each of the new PSPS Load/Merge machines. The layers have a minimum height greater than half the field of view. This is so that no single frame image spans more than 2 layers, 8 hence each layer must be >= 1.7 degrees. The spacing will be calculated from the full data set - it is more complex than just the density of sources on the sky - there is also the difference in the tables populated in the division between galactic and extragalactic regions. 8 9 9 10 [[Image(Kens1.png, 600px)]] … … 12 13 == Step 2 == 13 14 14 There are 8 DXlayer machines, each dxlayer machine is assigned to 4 contiguous Declination of the layers. Alternatively, the 8 DXlayers are subdivided into 4 layers in declination, making the 32 in total.15 There are 8 DXlayer machines, each dxlayer machine is assigned to 4 contiguous Declination of the layers. These do not have to be Alternatively, the 8 DXlayers are subdivided into 4 layers in declination, making the 32 in total. To aid in the convenience of making batches, the boundaries between these layers will be round to the nearest integer degree Declination. 15 16 16 17 [[Image(kens3.png, 600px)]] … … 19 20 == Step 3 == 20 21 21 For each set of 4 layers, called dxlayer[num], subdivide those to have segments of equal amount of data. This example here only has 8 segments per dxlayer[num]. Also, note that everything is divided in integer units of ra/dec (no fractional degrees) 22 For each set of 4 layers, called dxlayer[num], subdivide those to have segments of equal amount of data. This example here only has 8 segments per dxlayer[num]. Also, note that everything is divided in integer units of ra/dec (no fractional degrees). The division should be made by dividing the data in Right Ascension along a layer into X segments of approximately equal data size, where X is 24 for the ForcedWarp and Detections. These boundaries are also rounded off to the nearest integer degree of Right Ascension. 22 23 23 24 [[Image(kens2.png, 600px)]] … … 59 60 60 61 61
