Grand Reprocessing and first PSPS ingest Schedule (2011.06.28)


June 29 - We have finished Grand Reprocessing of the g band in the 14hr to 16 hr slice. This data is believed to be slightly compromised by the dark current model used for the older data.

June 29 - start GR of 14hr to 16 hr stripe in y band, followed by z band ~ 10 days of processing. For y and z band the noise is completely dominated by the noise in the sky background, and the dark current model does not affect the results.

July 10 - start MD04 reprocessing

before July 15 - start MD10 reprocessing, and creation of template. Then the incoming new MD10 data in OC142 will use this template and will be done on the new tesselation. The MD 07, 08, 09 will continue with old pixel size and tessellation so as not to interrupt transient program.

on or before July 15 - continue with GR with correct dark model in i, r, and g.

Schedule MD07 next as we learn how long reprocessing an MD field takes. Then in parallel with 3pi GR, proceed with MD05 and 06. We would like to be ready to reprocess MD 08 and 09 by the time each has finished for the year.

PSPS ingest and availability

June 30 - we will start ingesting of all the currently completed GR (LAP) DVO database NOW. When we have finished loading the g and y band, by approximately July 12, we will delete all the old data currently in the PSPS, and do the first merge/indexing of the completed GR data. Note the g,r,i,z,y from the initial LAP test data overlapping Stripe 82 will be there as well, but the g,r, and i data have higher noise from the dark current. This will take about two days during which PSPS will be offline.

July 15 - PSPS will be online and available for consortium use on July 15. From then on until next stage in photometry discussed below, we will be in ~normal ingest rate as the GR proceeds so by the end of July there should be all five bands from the 14 to 16 hour RA slice.


People should be aware that the ingested data has the same photometry issues as in the current IPP processing. Thus it is _not_ Data Release 1.0. It is data release 0.1. For details on the outstanding photometric issues see the PS1SC wiki DRAVG pages.

The g, r, and i band data in this release will have higher noise than they should. (The only 5 band data will be the initial LAP release that overlaps some of Stripe 82). It is believed this is a problem with the current dark current model being used for the old data we are re-processing. We will not save this pixel data, we will reproduce it with the correct dark model on the above schedule.

Towards Data Release 1.0

We will continue with GR as scheduled above, while working on the dark current model and outstanding photometry issues, including extended source photometry. (This does not include large galaxies, which are a special case for the next stage of development). When these are resolved (i.e "good enough") then the photometry analysis of the GR'd pixels will be redone. When this is complete we will do another deletion of what is in the PSPS and load the new results as Data Release 0.2. Ideally we would be ready to do this by the time the first 2hr slice of 5 band data is re-processed. Maybe this becomes Data Release 1.0, we hope so, but it may turn out to take more than one release before we "freeze" the IPP code for the year. These steps will be done in consultation with the Data Reduction and Verification Group, or DRAVG, with members at every institution.

Small Area Surveys

We hope to get the "Footprint" and "Small Area Surveys" observations soon as the weather improves. Processing these will become a priority as they become available, but this will take the place of the nightly ongoing processing, and won't impact GR or PSPS. We should be able to deliver Footprint and SAS stacks quickly after the observations are complete. The footprint survey is a single footprint with no dither of 3 years (60 total exposures) of 3pi data. The SAS is a 9x9 or 81 square degree survey to 3pi depth. Both are centered on MD09, and overlap SDSS Stripe 82.