PS1 Science Processing Status (20090705)

(up to IPP for PS1)

The PS1 Science Consortium has directed us to process a substantial amount of data for fast release, regardless of the data loss rate due to Magic mask. We have started with the 3 Medium Deep fields observed heavily since April: MD06, MD07, and MD08.

See Known data quality issues in the 2009.07.07 Release

Summary of data transfers for the 2009.07.07 Release

Some investigation of MDS image quality by Durham

Questions arising from the above MDS image analysis

from Photpipe - Harvard/JHU

Data in First Release

The first big batch of data from 3 Medium Deep fields should be available for download a bit later today (2009.07.07). This first set will consist of the June exposures from MD06, MD07, and MD08 (raw, chip, warp, diff) and the stacks used to generate the diffs. For this set of data, Paul Sydney has examined the magic results and certified the data to be released. There is a small number of images (7 or 8) which did not pass -- streaks leaked through -- and we will hold off on those until we can understand what happened (and mask them). The April and May data for these fields will be available soon -- the data has been processed, but we need a pass from Paul Sydney.

The June data consists of about 420 exposures. April and May together add up to about 300 more exposures.

For the purpose of managing the data transfer, Consortium sites need to register with the IPP team what portions of the data they would like to download.

Consortium sites may request data from people request any of the following 4 image sets, with the given typical sizes for the image and variance (does not exist for the raw data, of course):

Processing Stage Image Data per Exposure Data Products
Raw Data ~1.5GB Raw images + rawExp, rawImfile DB table entries
Chip (Detrended) ~3.6GB (images + variance) Detrend images, mask, variance, log, DB table entries
Camera (Calibrated) ~0.0GB (masks only) Calibrated detection files, logs, DB table entries
Warp ~6.0GB (images + variance) Warped images, mask, variance, log, DB table entries
Diff ~5.6GB (images + variance) Difference images, mask, variance, detection tables, log, DB table entries
Stack ~6.0GB per field per filter Stack images, mask, variance, detection tables, log, DB table entries

Except for the images and the variance images, the other data products (including the masks) are a small fraction of the image data value (< 5%).

The full June data set is about 420*(1.8 + 3.6 + 6 + 5.6) = 7TB. April + May will be another 5TB.

Institutions may register for subsets of the data, and may request (for example) to receive the detection files from the detrend chip stage, but no pixels, and all detections and pixels for the difference stage. At the moment, choices may be controlled independently for each MD field (and filter?).

A few other notes related to the data products for different stages:

* the detection tables generated at the chip stage (with names of the form are intermediate, uncalibrated data products. Are we distributing these still? The calibrated detections are rolled up into a single file (with many FITS extensions) with names of the form

* the detection tables from the warp stage are generated by transforming the detections from the chip stage, not by performing a full re-detection stage. The PSF is remeasured to determine the impact of warping on the image quality.

* the detection tables generated in the stack creation stage are (a) only going to limited depth and (b) only performing PSF analysis. These tables do not represent the full analysis of the stack images, but are used at this stage to check the output PSF model.

Summary of Data

field filter Nexp Ncam Nstack Stack Max(FWHM)
MD06 g.00000 25 25 15 5.7
"" r.00000 56 56 31 5.7
"" i.00000 105 101 44 5.0
"" z.00000 95 95 30 5.0
"" y.00000 14 14 14 5.0
"" all 295 291
MD07 g.00000 33 32 16 5.7
"" r.00000 46 46 21 6.5
"" i.00000 100 97 36 5.0
"" z.00000 76 76 17 5.2
"" y.00000 23 23 18 5.0
"" all 278 274
MD08 g.00000 36 35 19 6.0
"" r.00000 62 62 25 5.7
"" i.00000 81 80 20 5.0
"" z.00000 66 66 24 5.7
"" y.00000 8 8 8 5.0
"" all 253 251

The table above lists the number of exposures (Nexp) in each of the filters available for the three MD fields for the period April-June 2009. The column labeled Ncam lists the number of exposures processed at the camera stage (a small number failed on the chip analysis). The column labeled Nstack lists the number of exposures nominally used for each stack. Note that the exact number per stack depends on the skycell because of dithers and rotations. The last column gives the maximum seeing accepted for each stack (in pixels). Note that for MD06 and M08, since the y-band exposures all passed this limit, the actual max may be somewhat smaller than 5.0 pixels in these cases.

Data Requests

Institution Fields Filters Stages Mode Priority Notes
MPG MD06, MD07, MD08, grizy raw full 1 when possible
MPG MD06, MD07, MD08, grizy chip full 1
MPG MD06, MD07, MD08, grizy camera, warp full 2
MPG MD06, MD07, MD08, grizy stack, diff full 3
Durham, UK MD08 grizy stack full 1
Durham, UK MD07 grizy stack full 2
Durham, UK MD08 gz warp full 3
QUB, UK MD06, MD07, MD08 r warp,stack,diff full 1
QUB, UK MD06, MD07, MD08 griz diff cleaned
Edinburgh, UK MD06, MD07, MD08 ri stack,chip,raw,warp full
Edinburgh, UK MD06, MD07, MD08 grizy stack full
Edinburgh, UK MD06 grizy camera, stack full
Harvard,JHU MD06: skycell.077 grizy warp,stack,diff full 1 NGC 4258
Harvard MD08 grizy warp full 1 Overlaps with SDSS 82
Harvard MD08 grizy stack,diff full 1 Overlaps with SDSS 82
Harvard MD06, MD07, MD08 grizy warp full 2
Harvard MD06, MD07, MD08 grizy chip,camera full 2
Harvard MD06, MD07, MD08 grizy stack, diff full 3
JHU MD06, MD07, MD08 grizy chip,camera,warp,stk,diff cleaned 1
JHU MD08 gri warp,stk,diff full 2
JHU MD06 ri warp full 3
NCU, Taiwan MD07 grizy warp,diff,stack full 1
NCU, Taiwan MD06, MD08 giz warp,diff,stack cleaned 1
NCU, Taiwan MD06, MD08 y diff cleaned 1
NCU, Taiwan MD06, MD08 y warp,stack full 2
NCU, Taiwan MD06, MD08 r warp,diff,stack full 3

[Mode can be 'full': all products, or 'cleaned': only non-image products .]

A list of the various run ids can be found here Run ids for 200907 Release. A similar list for the stacks is here Stack ids for 200907 Release.

Launch Scripts

For the IPP Team's reference, the scripts used to queue the data for the analysis stages can be found in ~ipp/stdscience/stdsci.v1. For the chip analysis, there is as script for each field of with names like md06.v1.chip. These scripts select the exposures for each of the nights in this period and set up the labels and the workdirs (along with other defaults). The scripts used to launch the stack, diff, and magic runs have names like md08.v1.stk. The 'mode' variable for each of these scripts must be set manually to one of 'stack', 'diff', or 'magic'. For the moment, these scripts need to be run serially since they specify 'rerun' for the diff and magic stages. Eventually, we should convince ourselves that difftool and magictool correctly generate only a single run for a given input set, in which case these could be run automatically.

The stacks are all build using a good seeing subset of the data from the entire 3 month period. These selections were fairly subjective; in a future re-analysis pass, we can make a more educated choice for the inputs to maximize the S/N of the stack for point source detections (or other criterion as advised). Each of the stacks has a label of the form MD0n.200906.v1.


  • (82.9 KB) - added by heather 8 years ago. histograms of fwhm_major for MD08.200906.v0 and 200905.v0 for different filters