IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Ignore:
Timestamp:
Mar 16, 2017, 11:17:22 AM (9 years ago)
Author:
eugene
Message:

general eam revisions

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/release.2015/ps1.datasystem/datasystem.tex

    r40003 r40004  
    8888
    8989\section{Introduction}
    90 \label{sec: intro}
     90\label{sec:intro}
    9191
    9292This is the second in a series of seven papers describing the
    … …  
    156156
    157157This paper presents a description of the IPP data handling system.
    158 Section \ref{sec: subsystems} describes the major IPP subsystems that
     158Section \ref{sec:subsystems} describes the major IPP subsystems that
    159159underlie the main pipeline, providing a set of common interfaces and
    160160tools used at multiple stages.  The main processing stages of the
    161 pipeline are described in Section \ref{sec: stages}, although all
     161pipeline are described in Section \ref{sec:stages}, although all
    162162exposures may not necessarily pass through each of these stages.  The
    163163hardware systems that have done the processing for the PV3 data
    164 release are listed in Section \ref{sec: hardware}, with some details
     164release are listed in Section \ref{sec:hardware}, with some details
    165165on the scale of computing needed to reduce this large number of
    166 exposures.  Finally, Section \ref{sec: discussion} presents a
     166exposures.  Finally, Section \ref{sec:discussion} presents a
    167167discussion of some of the lessons learned in the creation of the IPP,
    168168and its utility in reducing data from other cameras and telescopes.
    … …  
    288288
    289289\section{IPP Data Processing Stages}
    290 \label{sec: stages}
     290\label{sec:stages}
    291291
    292292\subsection{Processing Database}
    293 \label{subsec: database}
     293\label{sec:database}
    294294
    295295A critical element in the Pan-STARRS IPP infrastructure is the
    … …  
    315315primary table which defines the conceptual list of processing items
    316316either to be done, in progress, or completed.  An associated secondary
    317 table lists the details of elements which have been processed.  Table
    318 \ref{tab: database schema} contains an outline of the database schema,
    319 showing the relations between tables organized by processing stage.
    320 As an example, one critical stage is the \ippstage{chip} processing
    321 stage, discussed below, in which the individual chips from an exposure
    322 are detrended and sources are detected.  Within the gpc1 database, the
    323 primary table called \ippdbtable{chipRun} in which each exposure has a
    324 single entry.  Associated with this table is the
    325 \ippdbtable{chipProcessedImfile} table, which contains one row for
    326 each of the (up to 60) chips associated with the exposure.  The
    327 primary tables, such as \ippdbtable{chipRun}, are populated once the
    328 system has decided that a specific item (e.g., an exposure) should be
    329 processed at that stage.  Initially, the entry is given a state of
    330 ``run'', denoting that the exposure is ready to be processed.  The
    331 low-level table entries, such as the \ippdbtable{chipProcessedImfile}
    332 entries, are only populated once the element (e.g., the chip) has been
    333 processed by the analysis system.  Once all elements for a given
    334 stage, e.g., chips in this case, are completed, then the status of the
    335 top-level table entry (\ippdbtable{chipRun}) are switched from ``run''
    336 to ``full''.
     317table (or set of tables) lists the details of elements which have been
     318processed.  Table \ref{tab: database schema} contains an outline of
     319the database schema, showing the relations between tables organized by
     320processing stage.  As an example, one critical stage is the
     321\ippstage{chip} processing stage (see \S\ref{sec:chip}) in which the
     322individual chips from an exposure are detrended and sources are
     323detected.  Within the gpc1 database, the primary table is called
     324\ippdbtable{chipRun} in which each exposure has a single entry.
     325Associated with this table is the \ippdbtable{chipProcessedImfile}
     326table, which contains one row for each of the chips
     327associated with the exposure (up to 60 for gpc1).  The primary tables, such as
     328\ippdbtable{chipRun}, are populated once the system has decided that a
     329specific item (e.g., an exposure) should be processed at that stage.
     330Initially, the entry is given a state of ``run'', denoting that the
     331exposure is ready to be processed.  The low-level table entries, such
     332as the \ippdbtable{chipProcessedImfile} entries, are only populated
     333once the element (e.g., the chip) has been processed by the analysis
     334system.  Once all elements for a given stage, e.g., chips in this
     335case, are completed, then the status of the top-level table entry
     336(\ippdbtable{chipRun}) are switched from ``run'' to ``full''.
    337337
    338338If the analysis of an element (e.g., the individual OTA chip)
    … …  
    357357data, dropping the failed chips from the rest of the analysis.  On the
    358358other hand, a \ippdbcolumn{fault} in one of the elements for a given
    359 stage will block any dependent stages from processing that item.  In
    360 this way, if such a temporary failure occurs, the system will not
    361 process an exposure through subsequent stages without the component
    362 that has failed temporarily.  Since many of the \ippdbcolumn{fault}s
    363 which occur are ephemeral, the processing stages are set up to
    364 occasional clear and re-try the faulted entries.  Thus, automatic
    365 processing is able to keep the data flowing even in the face of
    366 occasional network glitches or hardware crashes.
     359stage will block any successive stages which depend on that result
     360from processing that item.  In this way, if such a temporary failure
     361occurs, the system will not process an exposure through subsequent
     362stages without the component that has failed temporarily.  Since many
     363of the \ippdbcolumn{fault}s which occur are ephemeral, the processing
     364stages are set up to occasional clear and re-try the faulted entries.
     365Thus, automatic processing is able to keep the data flowing even in
     366the face of occasional network glitches or hardware crashes.
    367367
    368368\subsection{Summit copy}
    369 \label{subsec: summit copy}
     369\label{sec:summitcopy}
    370370
    371371As exposures are taken by the PS1 telescope \& GPC1 camera system, the
    … …  
    403403
    404404\subsection{Image Registration}
    405 \label{subsec: registration}
     405\label{sec:registration}
    406406
    407407Once the chips for an exposure have all been downloaded, the exposure
    … …  
    453453
    454454\subsection{Chip Processing}
    455 \label{subsec: chip}
     455\label{sec:chip}
    456456
    457457The science analysis of an exposure begins with the \ippstage{chip}
    … …  
    556556
    557557\subsection{Camera Calibration}
    558 \label{subsec: camera}
     558\label{sec:camera}
    559559
    560560After sources have been detected and measured for each of the chips,
    … …  
    611611
    612612\subsection{Fake Analysis}
    613 \label{subsec: fake}
     613\label{sec:fake}
     614\note{drop}
    614615
    615616The \ippstage{fake} stage was originally designed to do false source
    … …  
    628629
    629630\subsection{Image Warping}
    630 \label{subsec: warp}
     631\label{sec:warp}
    631632
    632633The \ippstage{warp} stage moves the data from a given exposure beyond
    … …  
    682683
    683684\subsection{Stack Combination}
    684 \label{subsec: stack}
     685\label{sec:stack}
    685686
    686687The skycell images generated by the \ippstage{warp} process are added
    … …  
    742743
    743744\subsection{Stack Photometry}
    744 \label{subsec: staticsky}
     745\label{sec:staticsky}
    745746
    746747Although images are generated in the \ippstage{stack} stage of the
    … …  
    801802
    802803\subsection{Forced Warp Photometry}
    803 \label{subsec: fullforce}
     804\label{sec:fullforce}
     805
     806\note{too much detail in this section; balance relative to psphot}
    804807
    805808Traditionally, projects which use multiple exposures to increase the
    … …  
    872875are excessively masked on a particular image are not used to model the
    873876PSF).  \note{this doesn't seem correct, as each warp is run
    874   independently.}  The PSF model is fitted to all of the known source
     877  independently. EAM: not true!}  The PSF model is fitted to all of the known source
    875878positions in the warp images.  Aperture magnitudes, Kron magnitudes,
    876879and moments are also measured at this stage for each warp.  Note that
    … …  
    881884warps.  When combined together, these low-significance measurements
    882885will result in a signficant measurement as the signal-to-noise
    883 increases by $\sqrt{N}$.
     886increases by the square root of the number of measurements.
    884887
    885888Upon completion of the forced photometry (for point sources as well as
    … …  
    899902\subsubsection{Forced Galaxy Models}
    900903\note{CZW: is this the appropriate place for this section?}
     904\note{too much detail in this section; balance relative to psphot}
    901905
    902906The convolved galaxy models are also re-measured on the
    … …  
    953957
    954958\subsection{Difference Images}
    955 \label{subsec: diff}
     959\label{sec:diff}
    956960Two of the primary science drivers for the Pan-STARRS system are the
    957961search hazardous asteroids and the search for Type Ia supernovae to
    … …  
    10201024
    10211025\subsection{DVO}
    1022 \label{subsec: DVO}
     1026\label{sec:DVO}
    10231027
    10241028The Pan-STARRS IPP uses an internal database system, distinct from the
    … …  
    10271031part of the astrometric and photometric calibration process.  This
    10281032database system, called the ``Desktop Virtual Observatory'' (DVO) was
    1029 developed originally for the LONEOS project, and used as part of the
     1033developed originally for the LONEOS project \citep{}, and used as part of the
    10301034CFHT Elixir system \citep{2004PASP..116..449M}.  The capabilities of
    10311035this databasing system have been somewhat expanded for the Pan-STARRS
    … …  
    10681072type of database table is stored as a separate file, or a collection
    10691073of files for table which are spatially partitioned.  The binary FITS
    1070 tables may be optionally compressed using the (to date) experimental
    1071 FITS binary table compression strategy outlined by \note{REF}.  In this
    1072 compression scheme, using a strategy similar to that used for FITS
    1073 image compression (\note{REF}), the data stored in the binary table is
    1074 compressed and stored in the 'HEAP' section of the FITS table.  In
    1075 brief, each column in the FITS table is compressed as one (or more)
    1076 blocks.  The standard fields which describe the data column format
    1077 (e.g., TFORM1) are replaced with columns which describe the location
    1078 and size of the compressed data in the HEAP section; the information
    1079 about the uncompressed data is moved to a field with 'Z' prepended
    1080 (e.g., ZFORM1) and an additional field is added to define the
    1081 compression algorithm (e.g., ZCTYP1).  The column names (e.g., TTYPE1)
    1082 and units (e.g., TUNIT1) are retained in their original form.  The
    1083 compression algorithm can treat the entire column as a single block of
    1084 data, or it may be broken into a number of chunks, each compressed in
    1085 turn (this must be the same for all columns).  Additional header
    1086 information is added to describe the block sizes and infomation needed
    1087 to describe the HEAP data section.  The compression algorithms
    1088 currently defined consist of the GZIP, RICE, PLIO, and HCOMPRESS
    1089 (REFS).  For GZIP, the compression algorithm may transpose the byte
    1090 order before compression: for floating point data of a similiar
    1091 dynamic range, this choice may allow for better compression as each
    1092 byte in the 4 or 8 byte floating point value is more likely to be
    1093 similar to the same byte in other rows than to the other bytes of the
    1094 same row value.  This option is called \code{GZIP_2} in the FITS
     1074tables are compressed using the (to date) experimental FITS binary
     1075table compression strategy outlined by \note{REF}.  Table compression
     1076is in general an option in DVO; for the PV3 database, the large data
     1077volume (70TB compressed) drove the decision to compress the tables.
     1078
     1079The FITS binary table compression scheme uses a strategy similar to
     1080that used for FITS image compression (\note{REF}).  The binary tabular
     1081data is compressed and stored in the 'HEAP' section of the FITS table
     1082extension, with pointers to the compressed data stored in the regular
     1083data section.  Each column in the FITS table is compressed as one (or
     1084more) blocks.  The standard header keywords which describe the data
     1085column format (e.g., TFORM1) are replaced with keywords which describe
     1086the location and size of the compressed data in the HEAP section; the
     1087information about the uncompressed data is moved to a keyword with 'Z'
     1088prepended (e.g., ZFORM1) and an additional field is added to define
     1089the compression algorithm (e.g., ZCTYP1).  The column names (e.g.,
     1090TTYPE1) and units (e.g., TUNIT1) are retained in their original form.
     1091
     1092The compression algorithm can treat the entire column as a single
     1093block of data, or it may be broken into a number of chunks, each
     1094compressed in turn (this must be the same for all columns).
     1095Additional header information is added to describe the block sizes and
     1096infomation needed to describe the HEAP data section.  The compression
     1097algorithms currently defined consist of the GZIP, RICE, PLIO, and
     1098HCOMPRESS (REFS).  For GZIP, the compression algorithm may transpose
     1099the byte order before compression: for floating point data of a
     1100similiar dynamic range, this choice may allow for better compression
     1101as each byte in the 4 or 8 byte floating point value is more likely to
     1102be similar to the same byte in other rows than to the other bytes of
     1103the same row value.  This option is called \code{GZIP_2} in the FITS
    10951104standard, as opposed to the standard order, \code{GZIP_1}.  The DVO
    10961105system can be set to specify the compression options for each column
    10971106in the tables.  In practice, we have chosen a default in which
    1098 floating point numbers used \code{GZIP_2}, character strings use
    1099 \code{GZIP_1}, integers use \code{RICE}. 
     1107floating point numbers use \code{GZIP_2}, character strings use
     1108\code{GZIP_1}, integers use \code{RICE}.
    11001109
    11011110\subsubsection{Sky Partition}
    11021111
    11031112DVO includes two major classes of database tables: those containing
    1104 information directly about astronomical objects in the sky and those
    1105 containing other supporting information.  The object-related tables
    1106 are partitioned on the basis of position in the sky: objects within a
     1113information about astronomical objects in the sky and those containing
     1114other supporting information.  The object-related tables are
     1115partitioned on the basis of position in the sky: objects within a
    11071116region bounded by lines of constant RA,DEC are contained in a specific
    11081117file.  The boundaries and the associated partition names are stored in
    … …  
    11111120(\ippdbcolumn{R\_MIN}, \ippdbcolumn{R\_MAX}, \ippdbcolumn{D\_MIN},
    11121121\ippdbcolumn{D\_MAX}), the name of the sky region, an ID
    1113 (\ippdbcolumn{INDEX}, equal to the sequence number of the region in the
    1114 table), and index entries to enable navigation within the table.  The
    1115 regions are defined in a hierarchical sense, with a series of levels
    1116 each containing a finer mesh of regions covering the sky.
     1122(\ippdbcolumn{INDEX}, equal to the sequence number of the region in
     1123the table), and index entries to enable navigation within the table.
     1124The regions are defined in a hierarchical sense, with a series of
     1125levels each containing a finer mesh of regions covering the sky.
    11171126
    11181127In the default used by the PV3 DVO, the partitioning scheme is based
    11191128on the one used by the Hubble Space Telescope Guide Star Catalog
    1120 files.  Level 0 is a single region covering the full sky.  Level 1
    1121 divides the sky in Declination into bands 7.5\degree\ high.  Level 2
    1122 subdivides these Declination bands in the RA direction, with spacing
    1123 related to the stellar density.  Level 3 divides these RA chunks into
    1124 4 - 8 smaller partitions.  This level exactly matches the HST GSC
    1125 layout, and uses the same naming convention to identify the
    1126 partitions: \code{n0000/0000}, etc. \note{more on the names?}.  Level
    1127 4 further divides these regions by a factor of 16.  In the
    1128 \ippdbtable{SkyTable}, a region at one level has a pointer to its
    1129 parent region (the one which contains it) and a sequence pointing to
    1130 its children (regions it contains).  The \ippdbtable{SkyTable} enables
    1131 fast lookups of the on-disk partitions which map to a specific
    1132 coordinate on the sky.  In general, a single DVO will have the full
    1133 sky represented with tables at a single level, though it is possible
    1134 for mixed levels to be used, this mode is not well tested.  For the
     1129files.  \note{add figure} Level 0 is a single region covering the full
     1130sky.  Level 1 divides the sky in Declination into bands
     11317.5\degree\ high.  Level 2 subdivides these Declination bands in the
     1132RA direction, with spacing related to the stellar density.  Level 3
     1133divides these RA chunks into 4 - 8 smaller partitions.  This level
     1134exactly matches the HST GSC layout, and uses the same naming
     1135convention to identify the partitions: \code{n0000/0000},
     1136etc. \note{more on the names?}.  Level 4 further divides these regions
     1137by a factor of 16.  In the \ippdbtable{SkyTable}, a region at one
     1138level has a pointer to its parent region (the one which contains it)
     1139and a sequence pointing to its children (regions it contains).  The
     1140\ippdbtable{SkyTable} enables fast lookups of the on-disk partitions
     1141which map to a specific coordinate on the sky.  In general, a single
     1142DVO will have the full sky represented with tables at a single
     1143level. Although it is possible for mixed levels to be used, this mode
     1144is not well tested and is avoided in the PV3 DVO database.  For the
    11351145PV3 master database, the partitioning at the \note{should this be
    11361146  4th?} 5th level results in \approx 150,000 regions to cover the full
    … …  
    11381148densest portions of the bulge contain at most \approx 300,000
    11391149astronomical objects in the database files, with an associated maximum
    1140 of 30 million measurements in these files.  With the compression
    1141 scheme described above, this makes the largest database files \approx
     1150of \approx 30 million measurements in these files.  With the compression
     1151scheme described above, the largest database files are \approx
    114211523GB, which can be loaded into memory in 30 seconds on the processing
    11431153machines that contain partition data.
     1154
     1155\note{is the use of the term 'partition host' consistent in this paper
     1156  and the calibration paper?}
    11441157
    11451158The DVO software system allows the tables which are partitioned across
    … …  
    11571170query operations will select the database information which matches
    11581171the query request (i.e., applying restrictions as defined) and return
    1159 to the master process the results.  The results from the various
     1172the results to the master process.  The results from the various
    11601173partition hosts are then merged into a single result by the master
    1161 process.  This parallelization is critical to querying and
    1162 manipulating the enormous database on a reasonable timescale.
     1174process.  When the parallel partitioning for a DVO instance is
     1175defined, the tables are randomly assigned to the partition hosts.  As
     1176a result, queries which span more than a single parition are likely to
     1177spread the I/O load across a large number of machines.  This
     1178parallelization is critical to querying and manipulating the enormous
     1179database on a reasonable timescale.
    11631180
    11641181\subsubsection{Astronomical Objects}
    … …  
    11681185\ippdbtable{SecFilt}.  These two tables specify the main average
    11691186properties of the astronomical object.  The \ippdbtable{Average} table includes the
    1170 astrometric information ($\alpha, \delta, \mu \alpha, \mu \delta,
     1187astrometric information ($\alpha, \delta, \mu_\alpha, \mu_\delta,
    11711188\pi$) and associated errors, data quality flags for each object, links
    11721189to the other tables, and a number of IDs, with one row for each
    … …  
    11981215
    11991216The individual measurements of the astronomical objects are carried in
    1200 the table \ippdbtable{Measure}.  This table lists the values measured
    1201 by \ippprog{psphot} for each \ippstage{chip}, \ippstage{warp}, or
    1202 \ippstage{stack} stage image.  This includes the instrumental magnitudes for
    1203 the PSF, aperture, and Kron photometry; raw position
     1217the table \ippdbtable{Measure}.  For measurements from PS1 in the PV3
     1218/ DR1 database, this would be values determined by \ippprog{psphot}
     1219for each \ippstage{chip}, \ippstage{warp}, or \ippstage{stack} stage
     1220image.  Measurements for other cameras processed by the IPP may also
     1221be included similarly in a DVO database.  Measurements from other
     1222sources, such as SDSS, 2MASS, or WISE, can also be included in this
     1223table (see \S\ref{sec:other.photometry}.
     1224
     1225The \ippdbtable{Measure} table includes the instrumental magnitudes
     1226for the PSF, aperture, and Kron photometry; raw position
    12041227(\ippdbcolumn{Xccd}, \ippdbcolumn{Yccd}) and second moments
    12051228(\ippdbcolumn{Mxx}, \ippdbcolumn{Myy}, \ippdbcolumn{Mxy}), along with
    … …  
    12071230(\ippdbcolumn{FWx}, \ippdbcolumn{FWy}, \ippdbcolumn{theta}).  Metadata
    12081231about the exposure that the measurement was derived from is also
    1209 include, such as the exposure time, the date \& time of the
     1232included, such as the exposure time, the date \& time of the
    12101233observation, airmass, azimuth, and \ippdbcolumn{photcode} information
    12111234specifying the filter.  The \ippdbtable{Measure} table also carries
    1212 the calibration magnitude offsts ($M_{\rm cal}$ and $M_{\rm flat}$,
     1235the calibration magnitude offsets ($M_{\rm cal}$ and $M_{\rm flat}$,
    12131236discussed below) and the astrometrically calibrated position.
    12141237Astrometric offsets for several systematic corrections discussed below
    … …  
    12161239photometry may have non-significant values, the table is somewhat
    12171240de-normalized in that it also carries instrumental flux values for the
    1218 PSF, aperture, and Kron photometry.
     1241PSF, aperture, and Kron photometry.  In this case, we have chosen to
     1242trade storage space for computing time.
    12191243
    12201244In the \ippdbtable{Measure} table, there are three fields which
    … …  
    12251249the \ippdbtable{Average} table the measurement belongs.  These two 32
    12261250bit fields can thus be combined into a single 64 bit ID unique for all
    1227 objects in the database.  In addition, the field
     1251objects in the database.  \note{PSPS IDs} In addition, the field
    12281252\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in
    12291253the \ippdbtable{Average} table of the associated object.  The
    12301254\ippdbtable{Measure} table may be unsorted, in which case it is slow
    12311255to find the measurements associated with a specific object (a full
    1232 table scan is required).  After the table is sorted, the
     1256table scan is required).  After the table is sorted and indexed, the
    12331257\ippdbcolumn{Measure} rows for a given object are grouped together.
    12341258In this case, the fields
    … …  
    12901314\subsubsection{Other Tables}
    12911315
    1292 Data from GPC1 (and other cameras processed by the IPP) are loaded
    1293 into DVO in units \code{smf} files generated by the \ippstage{camera}
    1294 calibration stage (see section \ref{subsec: camera} below).  As
    1295 described above, these files contain all measurements from a complete
    1296 exposure, with measurements from each chip grouped into separate FITS
    1297 tables.  When these measurements are loaded into the
    1298 \ippdbtable{Measure} and similar tables, a subset of the information
    1299 from the chip header is used to populated a row in the DVO
    1300 \ippdbtable{Image} table.  This table contains one row for each chip
    1301 known to DVO, with information such as the filter
    1302 (\ippdbcolumn{photcode}), the exposure time, the airmass, the
    1303 astrometric calibration terms, the photometric zero point, etc.  For
    1304 GPC1 and other mosaic cameras, an additional row is defined to carry
    1305 the projection and camera distortion elements of the astrometry model.
    1306 As chips are loaded into this table, they are assigned an internal ID
    1307 (a running sequence in the table).  Images may also be assigned an
    1308 external ID: in the case of the GPC1 images, this ID is defined by the
    1309 processing mysql database and is guaranteed to be unique within the
    1310 processing system.
     1316Measurements which are loaded into DVO may be associated with a
     1317specific image (such as the measurements for a single chip from the
     1318GPC1 camera) or they may not have such an association (such as
     1319measurements from 2MASS, WISE, or externally supplied reference
     1320measurements).  For data which is associated with an image, a subset
     1321of the information about that image (e.g., from the header of the FITS
     1322file) is used to populate a row in the DVO \ippdbtable{Image} table.
     1323This table contains one row for each chip image known to DVO, with
     1324information such as the filter (\ippdbcolumn{photcode}), the exposure
     1325time, the airmass, the astrometric calibration terms, the photometric
     1326zero point, etc.  For GPC1 and other mosaic cameras, an additional row
     1327is defined to carry the projection and camera distortion elements of
     1328the astrometry model.  As images are loaded into this table, they
     1329are assigned an internal ID (a running sequence in the table).  Images
     1330may also be assigned an external ID: in the case of the GPC1 images,
     1331this ID is defined by the processing mysql database and is guaranteed
     1332to be unique within the processing system.
     1333
     1334%% Data from GPC1 (and other cameras processed by the IPP) are loaded
     1335%% into DVO in units \code{smf} files generated by the \ippstage{camera}
     1336%% calibration stage (see section \ref{sec:camera} below).  As
     1337%% described above, these files contain all measurements from a complete
     1338%% exposure, with measurements from each chip grouped into separate FITS
     1339%% tables.  When these measurements are loaded into the
     1340%% \ippdbtable{Measure} and similar tables,
    13111341
    13121342Other tables are used to track information used by the calibration
    … …  
    13171347
    13181348\subsection{Addstar : DVO Ingest}
    1319 \label{subsec: addstar}
     1349\label{sec:addstar}
    13201350\note{CZW: This should be reviewed.}
    13211351
    … …  
    13511381
    13521382\subsection{Calibration Operations}
    1353 \label{subsec: calibration}
     1383\label{sec:calibration}
    13541384
    13551385\subsection{IPP to PSPS}
    1356 \label{subsec: ipp2psps}
     1386\label{sec:ipp2psps}
    13571387\note{Default to pointing to Flewelling et al 2017?}
    13581388
    13591389\subsection{PSPS Load \& Merge}
    1360 \label{subsec: psps}
     1390\label{sec:psps}
    13611391\note{Default as well to pointing to Flewelling et al 2017?}
    13621392
    … …  
    13641394
    13651395\subsection{Pantasks \& Parallel Processing}
    1366 \label{subsec: pantasks}
     1396\label{sec:pantasks}
     1397
     1398\note{this section needs to be re-written : pclient vs pcontrol vs pantasks}
    13671399
    13681400The actual processing of data is managed by the \ippprog{pantasks}
    … …  
    13771409
    13781410The \ippmisc{load} task for a particular stage queries the processing
    1379 database via an appropriate \ippmisc{ippTool} (see section \ref{subsec:
    1380   ipptools} below) for a list of jobs that are waiting to be run.
     1411database via an appropriate \ippmisc{ippTool} (see section \ref{sec:ipptools} below) for a list of jobs that are waiting to be run.
    13811412This task is executed on the host running the \ippprog{pantasks}
    13821413server, and only one of each type of \ippmisc{load} task is permitted to
    … …  
    14411472
    14421473\subsubsection{Stage automation}
    1443 \label{subsec: automation}
     1474\label{sec:automation}
    14441475\note{I'm not convinced this is the right place for it, but it felt like a natural extension of the ``advance''}.
    14451476
     1477\note{wording..}
    14461478Beyond the warp stage, there is no longer a single ``next'' stage into
    14471479which data can be queued.  Because of this, more robust methods are
    … …  
    14911523entries using whatever exposures are available if one has not yet been
    14921524constructed by the time the morning dark exposures are registered into
    1493 the database.
     1525the database. \note{wording}
    14941526
    14951527Automating the nightly processing is important, as it ensures that
    … …  
    14971529reducing the lag between an observation and the reduced data.  The
    14981530other processing task that requires automation is the reprocessing of
    1499 the entire $3\Pi$ survey, as the size of the dataset make it
     1531the entire $3\pi$ survey, as the size of the dataset make it
    15001532challenging to do manually.  To manage this, the ``large area
    15011533processing'' (LAP) task and script are used.  The first stage of this
    … …  
    15061538considered.  A \ippdbcolumn{projection\_cell} is a unit of sky defined
    15071539to be a square four degrees on each side which has a single tangent
    1508 plane projection \citep[][see]{waters2017}.  Once this entry is
    1509 defined, is is populated with exposures (stored in the
    1510 \ippdbtable{lapExp} table in the database), with any exposure located
    1511 within 5 degrees of the center of the projection cell included.  This
    1512 radius ensures that any exposure that overlaps the projection cell
    1513 will be included.  Once the exposures have been added, the other
    1514 exposures within the same sequence are checked to see if a
    1515 \ippstage{chip} stage entry has been generated, and if so, the
    1516 \ippdbcolumn{chip\_id} for that entry is saved into the
    1517 \ippdbtable{lapExp} as well.  This linkage ensures that each exposure
    1518 is only processed once.  If no entry is found, a new \ippstage{chip}
    1519 entry is queued for processing.  The task periodically checks the
    1520 status of the exposures in each \ippdbtable{lapRun} entry, and if they
    1521 have all completed the \ippstage{warp} stage, then a \ippstage{stack}
    1522 is queued for each skycell contained within the
     1540plane projection \citep[][see]{waters2017}.  \note{does waters2017
     1541  discuss RINGS.V3? if not, where?}  Once this entry is defined, is is
     1542populated with exposures (stored in the \ippdbtable{lapExp} table in
     1543the database), with any exposure located within 5 degrees of the
     1544center of the projection cell included.  This radius ensures that any
     1545exposure that overlaps the projection cell will be included.  Once the
     1546exposures have been added, the other exposures within the same
     1547sequence are checked to see if a \ippstage{chip} stage entry has been
     1548generated, and if so, the \ippdbcolumn{chip\_id} for that entry is
     1549saved into the \ippdbtable{lapExp} as well.  This linkage ensures that
     1550each exposure is only processed once.  If no entry is found, a new
     1551\ippstage{chip} entry is queued for processing.  The task periodically
     1552checks the status of the exposures in each \ippdbtable{lapRun} entry,
     1553and if they have all completed the \ippstage{warp} stage, then a
     1554\ippstage{stack} is queued for each skycell contained within the
    15231555\ippdbcolumn{projection\_cell}.
    15241556
    15251557\subsection{Nebulous}
    1526 \label{subsec: nebulous}
     1558\label{sec:nebulous}
    15271559Storing the large volume of data that is generated by the GPC1 camera
    15281560was recognized early in the Pan-STARRS project as a major concern.
    15291561The \ippprog{Nebulous} system was designed to organize this data.  The
    1530 main components of this system is a database storing the locations of
    1531 the files, with a Simple Object Access Protocol interface between the
    1532 database and the other IPP programs.  The actual files are stored on
    1533 NFS mounted partitions on a series of storage nodes in the IPP cluster
    1534 that can be accessed throughout the cluster.  This distribution of
    1535 files is useful to balance the disk I/O, as this parallelizes the
    1536 load.
     1562main components of this system are a database storing the locations of
     1563the files, with a Simple Object Access Protocol (SOAP) interface
     1564between the database and the other IPP programs \note{define / mention
     1565  http}.  The actual files are stored on a collection of computers
     1566with substantial disk partitions in the IPP cluster, shared within the
     1567cluster via NFS.  This distribution of files is useful to balance the
     1568disk I/O, as this parallelizes the load.
    15371569
    15381570The original design of \ippprog{Nebulous} was intended to aid in the
    … …  
    15461578reduced the degree to which the IPP processing is targeted.
    15471579
     1580\note{this is a critical paragraph and needs to be re-written to be
     1581  more accessible}
    15481582When a program creates a new file in \ippprog{Nebulous}, it supplies
    15491583an URI of the form \code{neb://HOST.VOLUME/PATH/FILENAME}.  The host
    15501584and volume specifiers are optional, and allow a file to be created on
    15511585a specific node.  The path and filename appear similar to a standard
    1552 full file location, and are used internally as the ``external id''.  A
    1553 storage object entry is then created in the database for this id, and
    1554 an instance of the file created on the specified node (or at random
    1555 from available nodes if left empty).  This instance is created in a
    1556 deterministic filename location.  The external id is hashed using the
    1557 SHA-1 function, and the first four hexadecimal digits of this hash are
     1586full file location, and are used internally as the ``external id''.
     1587\note{mention the nebulous schema before this?}  A storage object
     1588entry is then created in the database for this id, and an instance of
     1589the file created on the specified node (or at random from available
     1590nodes if left empty).  This instance is created in a deterministic
     1591filename location.  The external id is hashed using the SHA-1
     1592function, and the first four hexadecimal digits of this hash are
    15581593separated into two two-digit strings and used as the top and second
    15591594level directory location for the disk file.  The disk file is created
    … …  
    15641599\code{/data/HOST.VOLUME/nebulous/d5/d8/9244993440.PATH:FILENAME}.
    15651600This file naming structure is useful, as it duplicates database
    1566 contents on disk.
     1601contents on disk.  \note{rephrase}
    15671602
    15681603The storage volumes that contain the data on disk are defined in the
    … …  
    15781613volume should only be used as a backup volume (which accepts only
    15791614replicated copies), and the value of $5$ is used to indicate that the
    1580 volume is permanently unavailable, and should be ignored.
     1615volume is permanently unavailable, and should be ignored. \note{more
     1616  detail, more specific}
    15811617
    15821618In addition to this permanent table describing the volumes, a
    … …  
    15871623\ippprog{Nebulous} load balancing routines to prioritize those volumes
    15881624with large unused disk space before filling the volumes with only
    1589 small amounts remaining.  This table is regenerated every ten to
    1590 twenty minutes, after a scan of each of the volumes listed in the
     1625small amounts remaining.  This table is updated every ten to twenty
     1626minutes, after a scan of each of the volumes listed in the
    15911627\ippdbtable{volume} table.
    15921628
    15931629The final table controlling the operations of the \ippprog{Nebulous}
    15941630volumes is the \ippdbtable{cabinet} table, which organizes the
    1595 individual volumes into ``cabinets,'' a concept loosly based on the
     1631individual volumes into ``cabinets,'' a concept loosely based on the
    15961632physical arrangement of the storage servers in the data center.  These
    15971633cabinets are used to prevent the replication of a storage object
    15981634within a group of volumes where all instances of the object could be
    1599 taken off line by a single failure.  As the data center cabinets share
    1600 power supplies among the servers they contain, ensuring physical
    1601 distance between replicated copies is important to guarantee that a
    1602 temporary failure of one of these devices does not significantly
    1603 impact processing.
     1635taken off line by a single failure.  Since servers within a given
     1636cabinet in the data center share a common set of PDUs \note{define},
     1637it is important to ensure physical distance between replicated copies
     1638to guarantee that a temporary failure of one of the cabinet PDUs does
     1639not significantly impact processing.
     1640
     1641\note{need a paragraph or two on stats: how many objects, how many
     1642  instances?}
    16041643
    16051644\subsection{Datastore repositories}
    1606 \label{subsec: datastore}
     1645\label{sec:datastore}
    16071646
    16081647Transferring data between the IPP and other parts of the Pan-STARRS
    16091648system is generally accomplished via a ``datastore'', an http service
    1610 that exposes data in a common form.  One of the main datastores used
    1611 by the IPP is the one located at the summit.  This datastore exposes,
    1612 a list of the exposures obtained since the start of the PS1
    1613 operations.  Requests to this server may restrict to the latest by
    1614 time.  Each row in the listing includes basic information about the
    1615 exposure: an exposure identifier (e.g., o5432g0123o;
    1616 see~\ref{GPC1.names} for details), the date and time of the exposure,
    1617 the telescope commanded pointing, the filter and exposure time, and
    1618 the observation comment for that exposure.  The row also provides a
    1619 link to a listing of the chips associated with that exposure.  This
    1620 listing includes a link to the individual chip FITS files as well as
    1621 an md5 checksum.  Systems which are allowed access may download chip
    1622 FITS files via http requests to the provided links.
     1649that exposes data in a common form.  \note{add Isani / Hoblitt
     1650  reference?}  One of the main datastores used by the IPP is the one
     1651located at the summit.  This datastore exposes, a list of the
     1652exposures obtained since the start of the PS1 operations.  Requests to
     1653this server may restrict to the latest by time.  Each row in the
     1654listing includes basic information about the exposure: an exposure
     1655identifier (e.g., o5432g0123o; see~\ref{GPC1.names} for details), the
     1656date and time of the exposure, the telescope commanded pointing, the
     1657filter and exposure time, and the observation comment for that
     1658exposure.  The row also provides a link to a listing of the chips
     1659associated with that exposure.  This listing includes a link to the
     1660individual chip FITS files as well as an md5 checksum.  Systems which
     1661are allowed access may download chip FITS files via http requests to
     1662the provided links.
    16231663
    16241664The IPP also uses datastores to provide access to its own data
    … …  
    16331673
    16341674\subsection{ippTools and ippScripts}
    1635 \label{subsec: ipptools}
     1675\label{sec:ipptools}
    16361676
    16371677% \section{IPP Software Subsystems}
    1638 % \label{sec: subsystems}
     1678% \label{sec:subsystems}
    16391679
    16401680The IPP relies on a number of common libraries and programs to handle
    … …  
    17121752
    17131753\subsection{psLib and psModules}
    1714 \label{subsec: pslib}
     1754\label{sec:pslib}
    17151755
    17161756Underlying all of the analysis programs are the \ippmisc{psLib} and
    … …  
    17441784
    17451785\section{IPP Hardware Systems}
    1746 \label{sec: hardware}
     1786\label{sec:hardware}
    17471787
    17481788\note{what about psps hardware? mops hardware?}
    17491789
    17501790\subsection{Kihei Processing Cluster}
    1751 \label{subsec: kihei}
     1791\label{sec:kihei}
    17521792
    17531793The majority of all Pan-STARRS processing has been performed on the
    … …  
    17901830
    17911831\subsection{Los Alamos National Labs}
    1792 \label{subsec: LANL}
     1832\label{sec:LANL}
    17931833
    17941834In order to increase the processing rate for the $3\Pi$ PV3 data, we
    … …  
    18921932
    18931933\subsection{UH Cray Cluster}
    1894 \label{subsec: UH Cray}
     1934\label{sec:UHCray}
    18951935
    18961936In December 2014, the University of Hawaii installed a 178-compute
    … …  
    19181958
    19191959\section{Discussion}
    1920 \label{sec: discussion}
     1960\label{sec:discussion}
    19211961
    19221962\acknowledgments
    … …  
    19451985
    19461986\section{GPC1 Database Schema Outline}
    1947 \label{sec: database schema}
     1987\label{sec:database.schema}
    19481988
    19491989Table \ref{tab: database schema} provides a list of a majority of the
    … …  
    19541994tables for that stage listed together, along with the primary key
    19551995column that link the tables together.
     1996
     1997\note{logical or alphabetical sequence?  alignment is broken}
    19561998
    19571999\begin{center}
Note: See TracChangeset for help on using the changeset viewer.