Index: trunk/doc/release.2015/ps1.analysis/stages.tex
===================================================================
--- trunk/doc/release.2015/ps1.analysis/stages.tex	(revision 39821)
+++ trunk/doc/release.2015/ps1.analysis/stages.tex	(revision 39821)
@@ -0,0 +1,315 @@
+\documentclass[iop,floatfix]{emulateapj}
+% \documentclass[iop,floatfix,onecolumn]{emulateapj}
+% \pdfoutput=1
+
+\RequirePackage{color}
+\input{astro.sty}
+
+% online version may use color, but print version needs b/w
+\def\plotmode{col}
+%\def\plotmode{bw}
+
+%\def\plotext{pdf}
+\def\plotext{ps}
+
+%\def\picdir{/home/eugene/chipresid.20140404}
+\def\picdir{/data/pikake.2/eugene/chipresid.20140404}
+
+% Pick a terse version of the title here;
+\shorttitle{PS1 Data Processing Stages}
+\shortauthors{E.A. Magnier et al}
+\begin{document}
+\title{Pan-STARRS Data Processing Stages}
+
+% this is a crude trick to get the order of affiliations right.  These
+% names are used in the affiliations below.  The user needs to (1) set
+% the order and numbers to have the correct sequence in the author
+% list and (2) re-order the list at the bottom (and comment-out as needed)
+\def\IfA{1}
+\def\CfA{2}
+\def\MPIA{3}
+\def\Princeton{3}
+\def\USNO{4}
+\def\JHU{1}
+
+% This example has a first author from UH:
+\author{
+Eugene A. Magnier,\altaffilmark{\IfA}
+IPP Team,
+%PS Builder List
+% W.~S. Burgett,\altaffilmark{\IfA}
+% K.~C. Chambers,\altaffilmark{\IfA} 
+% L. Denneau,\altaffilmark{\IfA}
+% P. Draper,\altaffilmark{\DUR}
+% H.~A. Flewelling,\altaffilmark{\IfA}
+% T. Grav,\altaffilmark{\IfA}
+% J. N. Heasley,\altaffilmark{\IfA}
+% K. W. Hodapp,\altaffilmark{\IfA}
+% M. E. Huber,\altaffilmark{\IfA}
+% R. Jedicke,\altaffilmark{\IfA}
+% N. Kaiser,\altaffilmark{\IfA}
+% R.-P. Kudritzki,\altaffilmark{\IfA}
+% G. A. Luppino,\altaffilmark{\IfA}
+% R. H. Lupton,\altaffilmark{\Princeton}
+% E. A. Magnier,\altaffilmark{\IfA}
+% N. Metcalfe,\altaffilmark{\DUH}
+% D. G. Monet,\altaffilmark{\USNO}
+% J.~S. Morgan,\altaffilmark{\IfA}
+% P. M. Onaka,\altaffilmark{\IfA}
+% P.~A. Price,\altaffilmark{\Princeton}
+% C.~W. Stubbs,\altaffilmark{\CfA}
+% W.~E. Sweeney,\altaffilmark{\IfA}
+% J.~L. Tonry, \altaffilmark{\IfA}
+% R. J. Wainscoat,\altaffilmark{\IfA} and 
+% C. Z. Waters,\altaffilmark{\IfA}
+} % this bracket terminates author list
+
+% The ordering here should be sequential, matching the sequence in the list of authors:
+\altaffiltext{\IfA}{Institute for Astronomy, University of Hawaii, 2680 Woodlawn Drive, Honolulu HI 96822}
+% \altaffiltext{\CfA}{Harvard-Smithsonian Center for Astrophysics, 60 Garden Street, Cambridge, MA 02138}
+% \altaffiltext{\Princeton}{Department of Astrophysical Sciences, Princeton University, Princeton, NJ 08544, USA}
+% \altaffiltext{\USNO}{US Naval Observatory, Flagstaff Station, Flagstaff, AZ 86001, USA}
+% \altaffiltext{\JHU}{Department of Physics and Astronomy, Johns Hopkins University, 3400 North Charles Street, Baltimore, MD 21218, USA}
+% \altaffiltext{\MPIA}{Max Planck Institute for Astronomy, K\"onigstuhl 17, D-69117 Heidelberg, Germany}
+\begin{abstract}
+
+Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum
+bibendum nisi id tristique posuere. Duis eu mollis nulla. Maecenas est
+turpis, mattis tempor urna vitae, placerat rhoncus sem. Lorem ipsum
+dolor sit amet, consectetur adipiscing elit. Sed quis velit
+nisl. Aliquam erat volutpat. Cras lacinia, nisl tristique auctor
+molestie, dolor nulla rhoncus purus, ac accumsan nunc nunc ac
+nibh. Maecenas vitae mollis mauris. Ut sollicitudin pulvinar purus,
+eget luctus lorem tincidunt vitae. Vestibulum eu mattis neque. Nulla
+in tortor id urna dapibus gravida a vel leo.
+
+\end{abstract}
+
+% insert additional keywords as appropriate:
+\keywords{Surveys:\PSONE }
+
+% \section{INTRODUCTION}\label{sec:intro}
+
+\section{Processing Database}
+
+A critical element in the Pan-STARRS IPP infrastructure is the
+processing database.  This database, using the mysql database engine,
+tracks information about each of the processing stages.  It is used as
+the point of mediation of all IPP operations.  Processing stages
+within the IPP perform queries of the database to identify the data to
+be processed at a given stage.  As the processing for a particular
+stage is completed, summary information about the stage is written
+back to the database.  In this way, the database records this history
+of the processing, and also provides the information needed to
+successive processing stages to begin their own tasks.  
+
+The processing database is colloquially referred to as the `gpc1'
+database, since a single instance of the database is used to track the
+processing of images and data products related to the PS1 GPC1
+camera.  This same database engine also has instances for other
+cameras which the IPP has processed, e.g., GPC2, the test cameras TC1,
+TC3, the Imaging Sky Probe (ISP), etc.
+
+Within the processing database, the various processing stages are
+represented as a set of tables.  In general, there is a top level
+table which defines the conceptual list of processing items either to
+be done, in progress, or completed.  An associated table will list the
+specific details of elements which have been processed.  For example,
+one critical stage is the Chip processing stage, discussed below, in
+which the individual chips from an exposure are detrended and sources
+are detected.  Within the gpc1 database, there is a top-level table
+called `chipRun' in which each exposure has a single entry.
+Associated with this table is the `chipProcessedImfile' table, which
+contains one row for each of the (up to 60) chips associated with the
+exposure.  The top-level tables, such as chipRun, are populated once
+the system has decided that a specific item (e.g., an exposure) should
+be processed at that stage.  Initially, the entry is given a state of
+`run', denoting that the exposure is ready to be processed.  The
+low-level table entries, such as the chipProcessedImfile entries, are
+only populated once the element (e.g., the chip) has been processed by
+the analysis system.  Once all elements for a given stage, e.g., chips
+in this case, are completed, then the status of the top-level table
+entry (chipRun) will be switched from 'run' to 'done'.
+
+If the analysis of an element (e.g., chip) completed successfully,
+then the corresponding table row (e.g., chipProcessedImfile) is listed
+with a fault of 0.  If the analysis failed, then a non-zero fault is
+recorded.  An analysis which results in a fault is one in which the
+failure is thought to be temporary.  For example, if a computer had a
+network glitch and was unable to write out some of the result files,
+this would be an ephemeral failure which was not a failing of the
+data, but merely the processing system.  On the other hand, if the
+analysis failed because of a problem with the input data, this is
+noted by setting a non-zero value in a different table field,
+`quality'.  For example, if the chip analysis failed to discover any
+stars because the image was completely saturated, the analysis can
+complete successfully (fault = 0), but the `quality' field will be set
+to a non-zero value.  The various processing stages are able to select
+only the good (quality = 0) elements of a prior stage when choosing
+items for processing.  For example, the Camera calibration stage will
+only use data from chips with good quality data, dropping the failed
+chips from the rest of the analysis.  On the other hand, a fault in
+one of the elements for a given stage will block any dependent stages
+from processing that item.  In this way, if a glitch occurs and a chip
+from an exposure failed to be written to disk in the Chip stage, the
+system will not partially process the exposure with the rest of the
+chips.  Since many of the faults which occur are ephemeral, the
+processing stages are set up to occasional clear and re-try the
+faulted entries.  Thus, automatic processing is able to keep the data
+flowing even in the face of occasional network glitches or hardware
+crashes.
+
+\section{Download from Summit}
+
+As exposures are taken by the PS1 telescope \& camera system, the 60
+OTA CCDs are read out by the camera software system and each chip is
+written to disk on computers at the summit in the PS1 facility.  The
+chip images are written to a collection of machines in the PS1
+facility called the `pixel servers'.  After the images are written to
+disk, a summary listing of the information about the exposure and the
+chip images are written to an http server system called the
+`datastore'.  The datastore exposes, via http, a list of the exposures
+obtained since the start of the PS1 operations.  Requests to this
+server may restrict to the latest by time.  Each row in the listing
+includes basic information about the exposure: an exposure identifier
+(e.g., o5432g0123o; see~\ref{GPC1.names} for details), the date and
+time of the exposure, \note{etc}.  The row also provides a link to a
+listing of the chips associated with that exposure.  This listing
+includes a link to the individual chip FITS files as well as an md5
+CHECKSUM.  Systems which are allowed access may download chip FITS
+files via http requests to the provided links.
+
+During night-time operations, while the telescope is observing the sky
+and the camera subsystem is saving images to the pixel servers and
+adding their information to the datastore list, the IPP subsystem
+called `summitcopy' monitors the datastore in order to discover new
+exposures ready for download.  Once a new exposure has been listed on
+the datastore, summitcopy adds an entry of the exposure to a table in
+the processing database (summitExp).  This tells the summitcopy to
+look for the list of chips, which are then added to another table
+(summitImfile).  The summitcopy system then attempts to download the
+chips from the summit pixel servers with an http request.  As the chip
+files are downloaded, their md5 checksum values are calculated and
+compared with the value reported by the camera subsystem / datastore.
+Download failures are rare and marked as a non-zero fault, allowing for a
+manual recovery, rather than automatically rejecting the failed
+chips.  
+
+\section{Image Registration}
+
+Once chips for an exposure have all been downloaded, the exposure is
+ready to be registered.  In this context, `registration' refers to the
+process of adding them to the database listing of known, raw exposures
+(not to be confused with 'registration' in the sense of pixel
+re-alignment).  The result of the registration analysis is an entry
+for each exposure in the rawExp table, and one for each chip in the
+rawImfile table.  These tables are critical for downstream processing
+to identify what exposures are available for processing in any other
+stage.  In the registration stage, a large amount of descriptive
+metadata for each chip is added to the rawImfile table, some of which
+is extracted from the chip FITS file headers (e.g., RA, DEC, FILTER)
+and some of which is determined by a quick analysis of the pixels
+(e.g., mean pixel values, standard deviation).  The chip-level
+information is merged into a set of exposure-level metadata and added
+to the rawExp table entry.  The exposure-level metadata may be the
+same as any one of the chip, in a case where the values are duplicated
+across the chip files (e.g., the name of the telescope or the date \&
+time of the exposure), or it may be a calculation based on the values
+from each chip (e.g., average of the average pixel values).
+
+Unlike much of the rest of the IPP stage, the raw exposures may only
+have a single entry in the registration tables of the processing
+database tables (rawExp and rawImfile).
+
+\section{Chip Processing}
+
+The science analysis of an exposure begins with the processing of the
+individual chips, the Chip Processing stage.  This analysis step has
+two main goals: the removal of the instrumental signature from the
+pixel values (detrending) and the detection of the sources in the
+objects.  In the Chip stage, the individual chips are processed
+independently in parallel within the data processing cluster.  Within
+the processing computer cluster, most of the data storage resources
+are in the form of computers with large raids as well as substantial
+processing capability.  The processing system attempts to locate one
+copy of specific raw chips on pre-defined computers for each chip.
+The processing system is aware of this data localization and attempts
+to target the processing of a particular chip to the machine on which
+the data for that chip is stored.  The output products are then
+primarily saved back to the same machine.  This `targetted' processing
+was an early design choice to minimize the system wide network load
+during processing.  In practice, as computer disks filled up at
+different rates, the data has not been localized to a very high
+degree.  The targeted processing has probably reduced the network load
+somewhat but it has not been as critical of a requirement as
+originally expected.
+
+The Chip processing stage consists of: reading the raw image into
+memory, appyling the detrending steps (see~\note{Waters et al}),
+stiching the individual OTA cells into a single chip image, detection
+and characterization of the sources in the image (see~\note{Magnier et
+  al}), and output of the various data products.  These include the
+detrended chip image, variance image, and mask image, as well as the FITS
+catalog of detected sources.  The PSF model and background model are
+also saved, along with a processing log.  A selection of summary
+metadata describing the processing results are saved and written to
+the processing database along with the completion status of the
+process.  Finally, binned chip images are generated (on two scales,
+binned by 16 and 256 pixels) for use in the visualization system of
+the processing monitor tool.
+
+\section{Camera Calibration}
+
+After sources have been detected and measured for each of the chip,
+the next stage is to perform a basic calibration of the full exposure.
+This stage starts with the collection of FITS tables containing the
+instrumental measurements of the detected sources, primarily the
+positions ($x_{\rm ccd}, y_{\rm ccd}$) and the instrumental PSF
+magnitudes.  The data for all chips of an exposure are loaded by the
+analysis program.  The header information is used to determine the
+coordinates of the telescope boresite (RA, DEC, Position angle).
+These three coordinates are used, along with a model of the camera
+layout, to generate an initial guess for the astrometry of each chip.
+Reference star coordinates and magnitudes are loaded from a reference
+catalog for a region corresponding to the boundaries of the exposure,
+padded by a large fraction of the exposure diameter in case of a
+modest pointing error.  The guess astrometry is used to match the
+reference catalog to the observed stellar positions in the focal plane
+coordinate system.  Once an acceptable match is found, the astrometric
+calibration of the individual chips is performed, including a a fit to
+a single model for the distortion introduced by the camera optics.
+After the astrometic analysis is completed, the photometric
+calibration is determined using the final match to the reference
+catalog.  At this stage, pre-determined color terms may be included to
+convert the reference photometry to an appropriate photometric
+system.  For PS1, this is used to generate synthetic w-band photometry
+for areas where no PS1-based calibrated w-band photometry is
+available.  For more details, see \note{Magnier et al}.
+
+In addition to the astrometric and photometric calibrations, the
+Camera stage also generates the dynamic masks for the images.  The dynamic
+masks include masking for optical ghosts, glints, saturated stars,
+diffraction spikes, and electronic crosstalk.  The mask images
+generated by the Chip stage are updated with these dynamic masks and a
+new set of files are saved for the downstream analysis stages.
+
+The Camera stage also merges the binned chip images
+(see~\ref{sec:chip}) into single jpeg images of the entire focal
+plane.  These jpeg images can then be displayed by the process
+monitoring system to visualize the data processing.
+
+\begin{verbatim}
+Outline:
+Warp
+Stack
+Stack Photometry
+Forced Warp Photometry
+Forced Mean
+DVO Ingest
+Calibration
+IPP to PSPS
+PSPS Load & Merge
+Difference
+\end{verbatim}
+
+\end{document}