Ingest of LAP into PSPS

This wiki describes the ingest of LAP processing (specifically, LAP.ThreePi.20120706) into PSPS. There are several steps (not all completely finished)

Processing of LAP cam / skycal

This is currently ongoing. The first slice is to do between 18hr and 20hr (which contains kepler). This is called slice18, see the description of the slices below. Once all the projections cells within slice18 (or slicexx) are done, we start building the minidvodbs

Queue LAP cam / skycal

Once all the cam/skycals are processed within slice18, they will be queued to be built as a pair of minidvodbs:

• LAP.ThreePi.20120706.cam.slice18
• LAP.ThreePi.20120706.sky.slice18

These are not like typical minidvodbs in the sense that they will not be restricted to the 500 entries or < 1 day old rules. This may be changed, however depending on how things go.

Some minor tasks:

• add feature to addtool to queue by area of sky (this is apparently already done)
• add tweaks to the minidvodb script so that we don't create new minidvodbs every day or 500 entries

Build LAP cam skycal by slices

• once queued this is straightforward. Some tweaks may include to build the minidvodbs around the cluster (not just ipp004/ipp005)

Merge dvodb into big dvodb

• once the slices are built, we merge into a big parallel dvodb, and run the resort/averages/relphot commands on the slice in question or around the whole thing?

IPPtoPSPS creates batches based on slices

• this happens once the merge is done for that slice and the relphot/addstar commands are run at a minimum on that slice. To avoid boundary problems, ippToPSPS will be 3 degrees less on all edges (see slices below)
• the intent is to load neighboring slices. So, after slice18 is loaded, we load slice16, and the ranges within ipptopsps would be:
  • ra: 243 - 297 deg
  • dec: -33 to 75 deg
  • this is a combination of slice16 and slice18 with a 3 degree border, and i propose that we be very conservative with the cap.
• task: need ipptopsps to handle parallel dvodb

PSPS loads batches

• nothing to say here, conrad or thomas can fill in details

Slices

These are the following slices, labeled by Heather. The goal is to load connected slices (ie if slice18 is first then load either slice20 or slice 16 next)

slice name	load order	ra	dec	ra	dec	ipptopsps ranges ra	ipptopsps ranges dec
slice0	11*	0 to 2hr	-30 to 90 deg	0 to 30 deg	-30 to 90 deg	3 to 27 deg	-33 to 75 deg
slice2	12*	2 to 4hr	-30 to 90 deg	30 to 60 deg	-30 to 90 deg	33 to 57 deg	-33 to 75 deg
slice4	8*	4 to 6hr	-30 to 90 deg	60 to 90 deg	-30 to 90 deg	63 to 87 deg	-33 to 75 deg
slice6	7*	6 to 8hr	-30 to 90 deg	90 to 120 deg	-30 to 90 deg	93 to 117 deg	-33 to 75 deg
slice8	6*	8 to 10hr	-30 to 90 deg	120 to 150 deg	-30 to 90 deg	123 to 147 deg	-33 to 75 deg
slice10	5*	10 to 12hr	-30 to 90 deg	150 to 180 deg	-30 to 90 deg	153 to 177 deg	-33 to 75 deg
slice12	4*	12 to 14hr	-30 to 90 deg	180 to 210 deg	-30 to 90 deg	183 to 207 deg	-33 to 75 deg
slice14	3*	14 to 16hr	-30 to 90 deg	210 to 240 deg	-30 to 90 deg	213 to 237 deg	-33 to 75 deg
slice16	2*	16 to 18hr	-30 to 90 deg	240 to 270 deg	-30 to 90 deg	243 to 267 deg	-33 to 75 deg
slice18	1	18 to 20hr	20 to 90 deg	270 to 300 deg	20 to 90 deg	273 to 297 deg	23 to 75 deg	(this is kepler, and below is the galactic center)
slice18.2	13 *	18 to 20hr	-30 to 20 deg	270 to 300 deg	-30 to 20 deg	273 to 297 deg	-33 to 17 deg	(worry about the overlap between the 2 slice18s)
slice20	9*	20 to 22hr	-30 to 90 deg	300 to 330 deg	-30 to 90 deg	303 to 327 deg	-33 to 75 deg
slice22	10*	22 to 24hr	-30 to 90 deg	330 to 360 deg	-30 to 90 deg	333 to 357 deg	-33 to 75 deg

• means this is what Heather proposes- this can change prior to the actual load. The only certain one is that the first slice loaded is the one containing kepler (ie, slice18)

The idea is that Heather will load into the dvodb all the exposures (cam stage) with a boresite within the ra/dec range, and all the projection cells within the ra/dec range for a given slice. This means that certainly there will be data outside the edges of the slices within a minidvodb slice. This is ok.

First slice

First slice is slice18, but not entirely (we will only queue from 20 up):

The command to queue:

addtool -definebyquery -label LAP.ThreePi.20120706 -set_dvodb LAP.20120706 
-set_minidvodb_group LAP.slice18.cam -set_minidvodb -set_label LAP.slice18.cam 
-stage cam -uncensored -ra_min 270 -ra_max 300 -decl_min 20 -decl_max 90 -dbname gpc1

and then I added the minidvodb:

add.minidvodb    LAP.slice18.cam LAP.slice18.cam LAP.20120706 5 GPC1
add.label LAP.slice18.cam

the rest takes care of itself. I will add skycal probably tomorrow, once I verify that this part works fine.