Datastore Sky Tessellation

There is a desire to organize the datastores for PV3 into sky chunks that are of a similar size of data, organized by dxlayer and load merge machine. The intent is to always feed the PSPS a chunk of batches that add up to a similar size in bytes for each PSPS machine so there are no surprises. It also follows the new PSPS hardware design including the switches and network. See ​PSPS Hardware Plan C Over the course of the ingest and merge, the only thing that will significantly vary as we ingest across the sky is the cumulative size of the merged database. Each subsequent addition prior to merge, will be about the same size, and will be handled by a similar machine. Our greatest uncertainty at this stage is the scaling with the total size, not the scaling per chunk. This page describes a "datastore tessellation".

Step 1

Subdivide sky into the 32 declination layers. Each of the 32 layers has an equal amount of data in each layer (so, the thickness of the layers will vary, in order to increase/decrease the amount of data. These layers correspond to one each of the new PSPS Load/Merge machines. The layers have a minimum height greater than half the field of view. This is so that no single frame image spans more than 2 layers, hence each layer must be >= 1.7 degrees. The spacing will be calculated from the full data set - it is more complex than just the density of sources on the sky - there is also the difference in the tables populated in the division between galactic and extragalactic regions.

Step 2

There are 8 DXlayer machines, each dxlayer machine is assigned to 4 contiguous Declination of the layers. These do not have to be Alternatively, the 8 DXlayers are subdivided into 4 layers in declination, making the 32 in total. To aid in the convenience of making batches, the boundaries between these layers will be round to the nearest integer degree Declination.

Step 3

For each set of 4 layers, called dxlayer[num], subdivide those to have segments of equal amount of data. This example here only has 8 segments per dxlayer[num]. Also, note that everything is divided in integer units of ra/dec (no fractional degrees). The division should be made by dividing the data in Right Ascension along a layer into X segments of approximately equal data size, where X is 24 for the ForcedWarp and Detections. These boundaries are also rounded off to the nearest integer degree of Right Ascension.

Step 4

Each of these segments corresponds to a datastore. Each datastore is of the same size, and roughly maps to 1 dxlayer machine.

Final Construction

For OB/FO/DO/DF, because these are small batch sizes, we will subdivide each dxlayer[num] into 6 segments, giving a total of 48 datastores. ST/FG will have more than Object/Diff types of batches, so will divide each dxlayer[num] into 12 segments, giving a total of 96 datastores. P2/FW are the heaviest, and will be subdivided into 24 segments per dxlayer, for a total of 192 datastores.

The goal is to have PSPS load 8 datastores per merge (or possibly a multiple of that).

The names of the datastores will follow this format: PSPS_PV3_[batch]_dx[dxlayer]_seg[num], an example would be PSPS_PV3_OB_dx1_01

Expectations for the different batch types

type	dxlayers	segments per dxlayer	total datastores	load order
OB	8	6	48	1st
ST	8	12	96	2nd
P2	8	24	192	3rd
FW	8	24	192	3rd
FO	8	6	48	1st
FG	8	12	96	2nd
DO	8	6	48	last
DF	8	6	48	last