There is a desire to organize the datastores for PV3 into sky chunks that are of a similar size of data, organized by dxlayer and load merge machine. The intent is to always feed the PSPS a chunk of batches that add up to a similar size in bytes for each PSPS machine. This page describes Ken's suggested datastore tessellation, modified by Heather.

Step 1

Subdivide sky into the 32 declination layers. Each of the 32 layers has an equal amount of data in each layer (so, the thickness of the layers will vary, in order to increase/decrease the amount of data. These layers correspond to one each of the new PSPS Load/Merge machines.

Step 2

There are 8 DXlayer machines, each dxlayer machine is assigned to 4 contiguous Declination of the layers. Alternatively, the 8 DXlayers are subdivided into 4 layers in declination, making the 32 in total.

Step 3

For each set of 4 layers, called dxlayer[num], subdivide those to have segments of equal amount of data. This example here only has 8 segments per dxlayer[num]. Also, note that everything is divided in integer units of ra/dec (no fractional degrees)

Step 4

Each of these segments corresponds to a datastore. Each datastore is of the same size, and roughly maps to 1 dxlayer machine.

Final Construction

For OB/FO/DO/DF, because these are small batch sizes, we will subdivide each dxlayer[num] into 6 segments, giving a total of 48 datastores. ST/FG will have more than Object/Diff types of batches, so will divide each dxlayer[num] into 12 segments, giving a total of 96 datastores. P2/FW are the heaviest, and will be subdivided into 24 segments per dxlayer, for a total of 192 datastores.

The goal is to have PSPS load 8 datastores per merge (or possibly a multiple of that).

The names of the datastores will follow this format: PSPS_PV3_[batch]_dx[dxlayer]_seg[num], an example would be PSPS_PV3_OB_dx1_01

Expectations for the different batch types

type	dxlayers	segments per dxlayer	total datastores	load order
OB	8	6	48	1st
ST	8	12	96	2nd
P2	8	24	192	3rd
FW	8	24	192	3rd
FO	8	6	48	1st
FG	8	12	96	2nd
DO	8	6	48	last
DF	8	6	48	last