Changes between Initial Version and Version 1 of IppToPspsDatastorePV3Descriptions

Timestamp:

Nov 5, 2015, 5:31:04 PM (11 years ago)

Author:

heather

Comment:

--

IppToPspsDatastorePV3Descriptions

@@ +1 @@
+There is a desire to organize the datastores for PV3 into chunks that are of a similar size of data, organized by dxlayer. This describes the tessellation.
+
+== Step 1 ==
+
+Subdivide into the 32 layers. Each of the 32 layers has an equal amount of data in each layer (so, the thickness of the layers will vary, in order to increase/decrease the amount of data.
+
+
+
+== Step 2 ==
+
+There are 8 DXlayer machines, each dxlayer machine gets 4 of the layers.
+
+== Step 3 == 
+
+For each set of 4 layers, called dxlayer[num], subdivide those to have segments of equal amount of data. This example here only has 8 segments per dxlayer[num]. Also, note that everything is divided in integer units of ra/dec (no fractional degrees)
+
+== Step 4 ==
+
+Each of these segments corresponds to a datastore. Each datastore is of the same size, and roughly maps to 1 dxlayer machine.  
+
+== Final Construction ==
+
+
+For OB/FO/DO/DF, because these are small batch sizes, we will subdivide each dxlayer[num] into 6 segments, giving a total of 48 datastores. 
+ST/FG will have more than Object/Diff types of batches, so will divide each dxlayer[num] into 12 segments, giving a total of 96 datastores.
+P2/FW are the heaviest, and will be subdivided into 24 segments per dxlayer, for a total of 192 datastores.
+
+The goal is to have PSPS load 8 datastores per merge (or possibly a multiple of that).
+
+The names of the datastores will follow this format:  PSPS_PV3_[batch]_dx[dxlayer]_seg[num], an example would be PSPS_PV3_OB_dx1_01
+
+
+== Expectations for the different batch types ==
+
+
+|| type || dxlayers || segments per dxlayer || total datastores || load order ||
+|| OB   || 8        || 6                    || 48    ||  1st       || 
+|| ST   || 8        || 12                   || 96    ||  2nd       || 
+|| P2   || 8        || 24                   || 192   ||  3rd       || 
+|| FW   || 8        || 24                   || 192   ||  3rd       || 
+|| FO   || 8        || 6                    || 48    ||  1st       || 
+|| FG   || 8        || 12                   || 96    ||  2nd       || 
+|| DO   || 8        || 6                    || 48    ||  last       || 
+|| DF   || 8        || 6                    || 48    ||  last       || 
+
+
+
+
+
+
+
+