Changeset 40001
- Timestamp:
- Mar 16, 2017, 9:35:47 AM (9 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/release.2015/ps1.datasystem/datasystem.tex
r39975 r40001 176 176 submission and refereeing process.}} 177 177 178 \section{IPP Software Subsystems} 179 \label{sec: subsystems} 180 181 The IPP relies on a number of common libraries and programs to handle 182 various tasks that are shared between multiple stages of the 183 processing. These subsystems are described in this section, to 184 provide an introduction to these essential components that underlie 185 the rest of the pipeline. 178 \section{Overview of Pan-STARRS Data Processing} 179 180 The Pan-STARRS Data Analysis system contains many features to support 181 the wide range of activities: archiving and management of the raw and 182 processed image files; real-time nightly processing of images for 183 transient and moving object science; large-scale re-processing and 184 calibration to produce measurements for the science collaboration and 185 the wider public; \note{manual/specialized} image processing to 186 facilitate research and development of the analysis system itself; 187 distribution of the resulting data products to various consumers in a 188 variety of formats and modes. 189 190 The Pan-STARRS Data Analysis system is divided internally into several major 191 components: 192 \begin{itemize} 193 \item Summit : both the camera and observatory summit systems perform 194 data analysis tasks needed to support the on-going observations. 195 In this article, we focus on those aspects used by the off-summit 196 analysis stages. 197 \item Image Processing Pipeline (IPP) : this portion of the data 198 analysis system takes the data from raw pixels on the summit 199 computers to calibrated measurements of astronomical objects in an 200 internal databasing system. 201 \item Moving Object Processing System (MOPS) : this system is 202 responsible for linking individual detections of solar-system 203 objects together and determining the orbits. 204 \item PSPS : this system ingests the calibrated measurements from the 205 IPP, MOPS, and others and generates a high-availability database 206 with web-based interactions for public consumption. 207 \end{itemize} 208 The above set of analysis stages take place at the IfA within the 209 scope of responsibility of the Pan-STARRS Observatory. Within the 210 wider Pan-STARRS colloboration(s), additional data analysis operations 211 are performed to support science results. These collaboration-wide 212 analysis operations range from those which are tightly-coupled to the 213 Pan-STARRS Observatory system, such as the analysis of the transient 214 discovery teams and the public archive database at MAST, to those 215 which perform offline analysis for eventual ingest back into the 216 Pan-STARRS databases and archive. The latter category includes the 217 ubercal photometric analysis, the photo-z analysis, and the QSO / RR 218 Lyra search efforts. In addition, collaborations within the wider 219 Pan-STARRS community have implemented a variety of science-level 220 analyses of their own to support their science goals (e.g., M31 221 Cepheid search). 222 223 Figure~\ref{fig:analysis.elements} illustrates the many elements of 224 the Pan-STARRS data analysis system. This figure focuses on the data 225 analysis steps which occur within the Pan-STARRS observatory, with an 226 emphasis on the analysis, calibration, and database ingest stages. 227 The MOPS is described in detail by \cite{MOPS}, while the summit 228 systems are described by \note{REF?}. 229 230 Data analysis to support nighly science operations is driven by two 231 main goals: 1) rapid detection of the moving and transient sources to 232 enable recovery or follow-up with other telescopes. 2) regular 233 analysis of the images to monitor data quality and for use in 234 longer-timescale science projects. Not all of the analysis elements 235 listed in Figure~\ref{fig:analysis.elements} are used by the nightly 236 analysis system. Each of the data analysis stages are discussed in 237 detail below. In short, each image is processed independently to 238 correct for instrumental signatures and to detect the astronomical 239 sources (chip); astrometric and photometric calibrations are 240 determined (camera), and finally images are geometric transformed to a 241 common pixel representation (warp). Warped images may either be added 242 together (stack) or used in an image subtraction (diff). For nightly 243 science operations, images for certain fields such as the Medium Deep 244 survey fields (see \cite{}), are stacked together in nightly chunks, 245 providing deeper detection capability on short timescales. Depending 246 on the survey mode, difference images are generated for the nightly 247 stack images (vs a deep stack template) or for individual warp images. 248 In the later case, the warp images may be difference against another 249 warp from the same night or against a reference stack from the 250 appropriate part of the sky. 251 252 \note{need earlier mention of 3pi, MD, etc} 253 254 Pan-STARRS has performed several large-scale reprocessings of both the 255 Medium Deep and 3pi Survey data. For the 3pi Survey data, we identify 256 these large-scale reprocessings as PV1, PV2, and PV3 (we also define 257 the nightly science analysis of the data as PV0). For these 258 reprocessing stages, the standard steps of chip through warp, plus 259 stack and diff are performed, starting from raw data, using a single 260 homogenous version of the data analysis procedures. (PV2 was a 261 special case in which we started from the camera level products of 262 PV1). In addition to the analysis stages which are common with the 263 nightly processing, these large-scale reprocessing stages include 264 additional processing: a more detailed photometric analysis is 265 performed on the stacks, including morphological analysis appropriate 266 to galaxies. The results of the stack photometry analysis are used to 267 drive a forced-photometry analysis of the warp images. The data 268 products from the camera, stack photometry, and forced-warp photometry 269 analysis stages are ingested into the internal calibration database 270 (DVO, the Desktop Virtual Observatory) and used for photometric and 271 astrometric calibrations. 272 273 During the PS1 Science Consortium operations, data products were 274 provided to the consortium members from many different stages of the 275 analysis process. Data access by the PS1 Science Consortium members 276 was managed through a variety of mechanisms depending on the data 277 volume and type of data products desired. 278 Figure~\ref{fig:analysis.elements} illustrates some of these 279 connections. Access to small samples of imaging data was provided on 280 demand via the Postage Stamp server; access to large sets of 281 pre-defined raw and reduced data products was provided via the 282 Distribution and Publication systems. The interal calibration DVO 283 databases were provided at several stages via a separate DVO 284 distribution mechanism. For the first two large-scale reprocessings 285 (PV1 \& PV2), the data were ingested into the PSPS database system and 286 made available to the PS1SC community through a web portal based at 287 the IfA as well as the MAST portal. 288 289 \section{IPP Data Processing Stages} 290 \label{sec: stages} 186 291 187 292 \subsection{Processing Database} … … 260 365 processing is able to keep the data flowing even in the face of 261 366 occasional network glitches or hardware crashes. 367 368 \subsection{Summit copy} 369 \label{subsec: summit copy} 370 371 As exposures are taken by the PS1 telescope \& GPC1 camera system, the 372 data from the 60 OTA devices are read out by the camera software 373 wsystem and written to disk on a collection of computers at the summit 374 in the PS1 facility called ``pixel servers.'' After the images are 375 written to disk, a summary listing of the information about the 376 exposure and the chip images are added to the summit datastore. 377 378 During night-time operations, while the summit datastore is being 379 populated, the IPP subsystem called \ippstage{summitcopy} monitors the 380 datastores listed in the \ippdbtable{pzDatastore} table of the 381 database in order to discover new exposures ready for download. Once 382 a new exposure has been listed on the datastore, \ippstage{summitcopy} 383 adds an entry of the exposure to a table in the processing database 384 (\ippdbtable{summitExp}), indexed by an identifier that simply 385 increments the number of exposures announced by the summit, the 386 \ippdbcolumn{summit\_id}. This tells the \ippstage{summitcopy} system 387 to look for the list of chips, which are then added to another table 388 (\ippdbtable{summitImfile}). This system then attempts to download 389 the chips (registering the results of those operations into the 390 \ippdbtable{pzDownloadExp} and \ippdbtable{pzDownloadImfile} tables) 391 from the summit pixel servers via an http request. As the image files 392 are downloaded, their MD5 checksum values are calculated and compared 393 with the value reported by the summit datastore. Download failures 394 are rare and marked with a non-zero \ippdbcolumn{fault}, allowing for 395 a manual recovery, rather than automatically rejecting the failed 396 chips. Once all the components of the exposure have been downloaded, 397 they are further entered into the \ippdbtable{newExp} and 398 \ippdbtable{newImfile} tables, which index the exposures by 399 \ippdbcolumn{exp\_id}. This switch in index indicates that the 400 exposure has successfully been copied from the summit to the IPP 401 cluster, and that further processing is no longer dependent on outside 402 resources. 403 404 \subsection{Image Registration} 405 \label{subsec: registration} 406 407 Once the chips for an exposure have all been downloaded, the exposure 408 is ready to be registered. In this context, `registration' refers to 409 the process of adding them to the database listing of known, raw 410 exposures (not to be confused with 'registration' in the sense of 411 pixel re-alignment). The result of the registration analysis is an 412 entry for each exposure in the \ippdbtable{rawExp} table, and one for 413 each chip in the \ippdbtable{rawImfile} table. These tables are 414 critical for downstream processing to identify what exposures are 415 available for processing in any other stage. At the registration 416 stage, a large amount of descriptive metadata for each chip is added 417 to the \ippdbtable{rawImfile} table, the majority of which is 418 extracted from the chip FITS file headers (e.g., RA, DEC, FILTER) and 419 some of which is determined by a quick analysis of the pixels (e.g., 420 mean pixel values, standard deviation). The chip-level information is 421 merged into a set of exposure-level metadata and added to the 422 \ippdbtable{rawExp} table entry. The exposure-level metadata may be 423 the same as any one of the chip, in a case where the values are 424 duplicated across the chip files (e.g., the name of the telescope or 425 the date \& time of the exposure), or it may be a calculation based on 426 the values from each chip (e.g., average of the average pixel values). 427 428 Unlike much of the rest of the IPP stage, the raw exposures may only 429 have a single entry in the registration tables of the processing 430 database tables (\ippdbtable{rawExp} and \ippdbtable{rawImfile}). 431 432 For GPC1, the image registration stage is also the stage at which the 433 \ippprog{burntool} analysis is run. This analysis is more completely 434 described in \citet{waters2017}. In brief, the \ippprog{burntool} 435 program identifies bright sources on the image, and identifies 436 persistence trails that result from the incomplete transfer of charge. 437 As this charge can leak out in subsequent exposures, the burntool 438 analysis is run sequentially on the exposures, based on the 439 observation date and time listed in the headers, with the results 440 stored in an text table. As a result of the sequential nature of this 441 analysis, the registration of exposures is blocked until the 442 \ippprog{burntool} has been run on the previous exposures. 443 444 Once the registration process has finished, new science exposures that 445 have an \ippdbcolumn{obs\_mode} value that indicates they are part of 446 a particular science survey are automatically launched into the 447 science analysis by defining entries for the \ippstage{chip} 448 processing stage, as described above. This analysis can be relaunched 449 multiple times, such as for the large scale PV3 reprocessing. 450 However, this automatic process ensures the shortest time between 451 observation and analysis, which is particularly important in the 452 search for transient sources. 453 454 \subsection{Chip Processing} 455 \label{subsec: chip} 456 457 The science analysis of an exposure begins with the \ippstage{chip} 458 stage, which operates on the individual OTA image files. This 459 analysis step has two main goals: detrending the image to remove the 460 instrumental signature from the pixel values, and the detection of 461 astronomical sources in the objects. Based on the entry the 462 \ippdbtable{chipRun} primary table defining the processing details 463 (with the \ippdbcolumn{state} column indicating it needs processing), 464 and the associated information listed in the \ippdbtable{rawImfile}, 465 jobs can be spawned for each component OTA. The \ippprog{pantasks} 466 environment managing the jobs attempts to target the processing host 467 to one that should host the OTA, to reduce number of operations done 468 on remote data. In practice, this targeted processing has not had as 469 large of an effect as was originally intended, as the data volume has 470 reduced the ability of any one node to reliably contain a particular 471 OTA. The targeted processing has probably reduced the network load 472 somewhat but it has not been as critical of a requirement as 473 originally expected. 474 475 %% In the \ippstage{chip} stage, 476 %% the individual OTA image files are processed independently in parallel 477 %% within the data processing cluster. \note{move this to kihei 478 %% discussion?} Within the processing computer cluster, most of the 479 %% data storage resources are in the form of computers with large raids 480 %% as well as substantial processing capability. The processing system 481 %% attempts to locate one copy of specific raw registered data on 482 %% pre-defined computers that have been set as storage targets for that 483 %% OTA. The processing system is aware of this data localization and 484 %% attempts to target the processing for each OTA to the machine on which 485 %% the data for that detector is stored. The output products are then 486 %% primarily saved back to the same machine. This `targetted' processing 487 %% was an early design choice to minimize the system wide network load 488 %% during processing. In practice, as computer disks filled up at 489 %% different rates, the data has not been localized to a very high 490 %% degree. 491 492 The actual image processing is performed by the \ippprog{ppImage} 493 program. This program reads the raw data into memory and applies the 494 detrend corrections \citep[see][]{waters2017} to each cell in the OTA 495 (which are stored as different extensions in the FITS file format), 496 and then mosaics the cells into a single contiguous \ippstage{chip} 497 stage image. This step also creates in memory additional images to 498 hold the mask data, which indicates which pixels may not be valid, and 499 the variance image, constructed as the Poissonian noise on the number 500 of electrons detected based on the original pixel value and the 501 detector gain. A background model is then fit across the image and 502 subtracted to remove the expected contribution from the sky 503 \citep[see][]{waters2017} for details. 504 505 With the image calibration procedure finished, object identification 506 and photometry can be performed. Although this can be done using a 507 stand alone program, \ippprog{psphot}, the underlying functions are 508 contained in a library that allows \ippprog{ppImage} to directly do 509 this analysis, removing the need to write out and re-read the image 510 data. The details of the detection and characterization of the 511 sources in the image are provided in \citet{magnier2017b}. 512 513 The results of the image processing are then written to disk, 514 including the science, mask, and variance images, the background model 515 subtracted, the PSF model used in the photometry process, and a FITS 516 catalog of detected sources. Additional binned images of the full OTA 517 are also saved, providing $16\times{}16$ and $256\times{}256$ pixel 518 binning scales for quick visualization. The processing log and a 519 selection of summary metadata describing the processing results are 520 also written to disk. This metadata is used to populate a row in the 521 \ippdbtable{chipProcessedImfile} table (linked to the 522 \ippdbtable{chipRun} entry by a shared \ippdbcolumn{chip\_id} value) 523 to indicate that the processing of this OTA is complete. 524 525 As each OTA is processed independently of the others across a number 526 of computers, the \ippprog{pantasks} managing the jobs periodically 527 runs an \ippmisc{advance} task that checks that the number of rows in 528 \ippdbtable{chipProcessedImfile} with \ippdbcolumn{fault} equal to 529 zero matches the associated number of rows in \ippdbtable{rawImfile}. 530 If this condition is met, than all processing for that exposure is 531 finished, and the \ippdbcolumn{state} field is set to ``full''. If 532 the \ippdbtable{chipRun}.\ippdbcolumn{end\_stage} field is set to 533 \ippstage{chip}, then no further action is taken. However, this field 534 is usually set to a subsequent stage (most often \ippstage{warp}), 535 then an entry for this exposure is added to the \ippdbtable{camRun} 536 table, and processing continues. 537 538 %% The \ippstage{chip} processing stage consists of: reading the raw image into 539 %% memory, applying the detrending steps \citep[see][]{waters2017}, 540 %% stiching the individual OTA cells into a single chip image, detection 541 %% and characterization of the sources in the image 542 %% \citep[see][]{magnier2017b}, and output of the various data products. 543 %% These include the detrended chip image, variance image, and mask 544 %% image, as well as the FITS catalog of detected sources. The PSF model 545 %% and background model are also saved, along with a processing log. A 546 %% selection of summary metadata describing the processing results are 547 %% saved and written to the processing database along with the completion 548 %% status of the process. Finally, binned chip images are generated (on 549 %% two scales, binned by 16 and 256 pixels) for use in the visualization 550 %% system of the processing monitor tool. \note{describe elsewhere?} 551 552 %% The database structure for the \stage{chip} stage mimics that of raw 553 %% data, with a \ippdbtable{chipRun} characterizing the processing of a 554 %% single exposure, mapping to a set of \ippdbtable{chipProcessedImfile} 555 %% entries for each OTA via a common \ippdbcolumn{chip\_id}. 556 557 \subsection{Camera Calibration} 558 \label{subsec: camera} 559 560 After sources have been detected and measured for each of the chips, 561 the next stage is to perform a basic calibration of the full exposure 562 in the \ippstage{camera} stage. This runs as a single job for the 563 entire exposure, passing the collection of FITS table catalogs 564 generated from each OTA in the \ippstage{chip} stage to the 565 \ippprog{psastro} program. Although the full catalog is loaded, the 566 calibration primarily concerns the positions ($x_{\rm ccd}, y_{\rm 567 ccd}$) and the instrumental PSF magnitudes. The header information 568 in these catalogs is used to determine the coordinates of the 569 telescope boresite (RA, DEC, position angle). These three coordinates 570 are used, along with a pre-determined model of the OTA layout within 571 the camera, to generate an initial guess for the astrometry of each 572 chip. Reference star coordinates and magnitudes are loaded from a 573 reference catalog for a region corresponding to the boundaries of the 574 exposure, padded by a large fraction (25\%) of the exposure diameter 575 to help guarantee a solution in the case of a modest pointing error. 576 The guess astrometry is used to match the reference catalog to the 577 observed stellar positions in the focal plane coordinate system. Once 578 an acceptable match is found, the astrometric calibration of the 579 individual chips is performed, including a fit to a single model for 580 the distortion introduced by the camera optics. After the astrometic 581 analysis is completed, the photometric calibration is determined using 582 the final match to the reference catalog. At this stage, 583 pre-determined color terms may be included to convert the reference 584 photometry to an appropriate photometric system. For PS1, this is 585 used to generate synthetic w-band photometry for areas where no 586 PS1-based calibrated w-band photometry is available. For more 587 details, see \cite{magnier2017c}. The result of these calibrations is 588 stored as a single multi-extension FITS table containing the results 589 from each OTA as a separate extension. 590 591 In addition to the astrometric and photometric calibrations, the 592 \ippstage{camera} stage also generates the dynamic masks for the 593 images. These include masking for optical ghosts, glints, saturated 594 stars, diffraction spikes, and electronic crosstalk. The mask images 595 generated by the \ippstage{chip} stage are updated with these dynamic 596 masks and a new set of files are saved for the downstream analysis 597 stages. The \ippstage{camera} stage also merges the binned chip 598 images (see~\ref{sec:chip}) into single jpeg images of the entire 599 focal plane. These jpeg images can then be displayed by the process 600 monitoring system to visualize the data processing. 601 602 Again, summary metadata is saved to disk as well, and the results 603 listed therein are used to populate a row in the 604 \ippdbtable{camProcessedExp} database table. As the full exposure is 605 processed all at once, this update also updates the associated 606 \ippdbtable{camRun} entry, linked by the \ippdbcolumn{cam\_id}. As 607 with the \ippstage{chip} stage, the 608 \ippdbtable{camRun}.\ippdbcolumn{end\_stage} is for a subsequent 609 stage, an appropriate entry is added to the \ippdbtable{fakeRun} 610 table. 611 612 \subsection{Fake Analysis} 613 \label{subsec: fake} 614 615 The \ippstage{fake} stage was originally designed to do false source 616 injection and recovery, in order to determine the detection efficiency 617 of sources on the exposure. However, early in the design of the IPP, 618 this task was moved to the rest of the photometry analysis done at the 619 \ippstage{chip} stage. Removing the stage would require significant 620 changes to the database schema. As a result, this conveniently named 621 stage generally does no actual data processing, and consists mainly of 622 database operations to move the exposure on to the \ippstage{warp} 623 stage. The operations mimic the \ippstage{chip} stage, with 624 individual jobs run for each OTA that update rows in the 625 \ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that 626 updates the \ippdbtable{fakeRun} table and promotes the exposure to 627 the next stage by adding a row to the \ippdbtable{warpRun} table. 628 629 \subsection{Image Warping} 630 \label{subsec: warp} 631 632 The \ippstage{warp} stage moves the data from a given exposure beyond 633 away from being camera specific and towards a uniform sky oriented 634 arrangement. There are a number of ``tessellations'' defined and used 635 by the IPP to define the extent and scaling of images on this uniform 636 arrangement. A tessellation can be defined for a limited region, such 637 as M31 or other fields of particular interest that can be well 638 described by a single tangent plane projection, or for larger regions 639 which have multiple projection centers. For the $3\Pi$ survey, the 640 \ippmisc{RINGS.V3} tessellation was used that used projection centers 641 spaced every four degrees in both RA and DEC, with $0\farcs{}25$ 642 pixels. These projections are further broken down into ``skycells'' 643 that form a $10\times{}10$ grid within the projection, with an overlap 644 region of 60" between adjacent skycells to ensure that objects are not 645 split on all images. 646 647 These tessellations are stored in the DVO format, with 648 \ippdbtable{SkyTable} entries defining the projection centers and 649 image boundaries for all the skycells. The first step of the 650 \ippstage{warp} stage is determining which skycells overlap with the 651 input exposure. These overlaps are determined by the 652 \ippprog{dvoImageOverlaps} program, which compares the astrometrically 653 calibrated catalog from the \ippstage{camera} stage to the 654 \ippdbtable{SkyTable} entries. The output of this command is used to 655 populate the \ippdbtable{warpSkyCellMap} table in the database, which 656 contains a row for each skycell and OTA that overlap. This results in 657 more rows than there are OTAs, as each skycell can contain 658 contributions from multiple OTAs. 659 660 Once this mapping has been defined, jobs to construct each skycell are 661 run, passing the \ippstage{camera} stage catalog and the 662 \ippstage{chip} stage images (including the variance images and the 663 updated masks) to the \ippprog{pswarp} program. For details on the 664 warping algorithm, see \cite{waters2017}. The output of this program 665 are the geometrically transformed images containing all input pixels 666 warped to the common skycell pixel grid, which can subsequently be 667 used for stacking and difference image analysis. The image, mask, and 668 variance generated at this stage will be available from the image 669 extraction tools at the MAST archive at STScI as part of the DR2 data 670 release. A catalog is also generated containing the locations of 671 sources from the input catalog that fall within area of the 672 \ippstage{warp}. 673 674 When the jobs have completed, an entry for the skycell is added to the 675 \ippdbtable{warpSkyfile} database table, linked to the 676 \ippdbtable{warpRun} entry by a common \ippdbcolumn{warp\_id}. An 677 \ippmisc{advance} task again checks that all potential skycells have 678 been generated. At this point, the direct promotion of exposures from 679 one stage to the next stops, as the logic for matching exposures for 680 combination is more complicated than simply adding a single entry (as 681 discussed above). 682 683 \subsection{Stack Combination} 684 \label{subsec: stack} 685 686 The skycell images generated by the \ippstage{warp} process are added 687 together to make deeper, higher signal-to-noise images in the 688 \ippstage{stack} stage. These stacked images also fill in coverage 689 gaps between different exposures, resulting in an image of the sky 690 with more uniform coverage than a single exposure. 691 692 In the IPP processing, stacks may be made with various options for the 693 input images. During nightly science processing, the 8 exposures per 694 filter for each Medium Deep field are combined into a set of stacks 695 for that field. These so-called `nightly stacks' are used by the 696 transient survey projects to detect faint supernovae, among other 697 transient events. For the PV3 $3\pi$ analysis, all images in each 698 filter from the observations for this survey were stacked together to 699 generate a single set of images with $\sim 10 - 20\times$ the exposure 700 of the individual survey exposures. 701 702 For the PV3 processing of the Medium Deep fields, stacks have been 703 generated for the nightly groups and for the full depth using all 704 exposures, producing ``deep stacks''. In addition, a 'best seeing' 705 set of stacks have been produced \note{using image quality cuts to be 706 described}. We have also generated out-of-season stacks for the 707 Medium Deep fields, in which all image not from a particular observing 708 season for a field are combined into a stack. These later stacks are 709 useful as deep templates when studying long-term transient events in 710 the Medium Deep fields as they are not (or less) contaminated by the 711 flux of the transients from a given season. 712 713 When a given set of \ippstage{stack} stage are defined, exposures with 714 existing \ippstage{warp} entries that match the filter, position, and 715 other criteria such as seeing are grouped by their skycell. An entry 716 is then added for each skycell in the \ippdbtable{stackRun} table, 717 with the \ippdbcolumn{warp\_id} entries for the exposures added to the 718 \ippdbtable{stackInputSkyfile} table, linked to the 719 \ippdbtable{stackRun} entry by the \ippdbcolumn{stack\_id} field. 720 This defines the mapping for which exposures contribute to the 721 \ippstage{stack}. This breaks exposures into single skycells, but as 722 adjacent \ippstage{stack} skycells may contain inputs from different 723 exposures, there is no simple way to group the processing at the 724 \ippstage{stack} stage into exposures. 725 726 The \ippstage{stack} jobs pass the information about the input images 727 and catalogs to the \ippprog{ppStack} program, which performs the 728 image combinations. See~\cite{waters2017} for details on the stack 729 combination algorithm. In addition to the standard image, mask, and 730 variance produced at other stage, additional images are constructed 731 with information about the contributions to each pixel. A number 732 image contains the number of input exposures used for each pixel, 733 along with an exposure time map, and a weighted exposure time map that 734 scales the exposure time based on the relative variance of each input. 735 These images for the $3\Pi$ analysis are currently available from the 736 MAST image extraction tools at STSci. 737 738 Upon completing the generation of these images, a row is added to the 739 \ippdbtable{stackSumSkyfile} table with statistics about 740 \ippstage{stack} processing. As this completes all processing for the 741 entry, no \ippmisc{advance} job is required. 742 743 \subsection{Stack Photometry} 744 \label{subsec: staticsky} 745 746 Although images are generated in the \ippstage{stack} stage of the 747 IPP, the source detection and extraction analysis of those images is 748 deferred to the \ippstage{staticsky} stage. This separation is 749 maintained because the photometry analysis of the \ippstage{stack} 750 images is performed on all 5 filters simultaneously. By deferring 751 this analysis, the processing system may also decouple the generation 752 of the pixels from the source detection. This makes the sequencing of 753 analysis somewhat easier and less subject to blocks due to a failure 754 in the stacking analysis. Similar to the \ippstage{stack} stage, an 755 entry is created in the \ippdbtable{staticskyRun} table, linked to a 756 series of rows in the \ippdbtable{staticskyInput} table by a common 757 \ippdbcolumn{sky\_id}, each of which also contains the appropriate 758 \ippdbcolumn{stack\_id} entries for the skycell under consideration. 759 760 The input images are passed to the \ippprog{psphotStack} program, 761 which does the analysis. The stack photometry algorithms are 762 described in detail in \cite{magnier2017b}. In short, sources are 763 detected in all 5 filter images down to the $5\sigma$ significance. 764 The collection of detected sources is merged into a single master 765 list. If a source is detected in at least two bands, or only in 766 \yps{} band, then a PSF model is fitted to the pixels of the other 767 bands in which the source was not detected. This forced photometry 768 results in lower significance measurements of the flux at the 769 positions of objects which are thought to be real sources, by virtue 770 of triggering a detection in at least two bands. The relaxed limit 771 for \yps{} band is included to allow for searches of \yps{} dropout 772 objects: it is known that faint, high-redshift quasars may be detected 773 in \yps{} band only. Sources detected only in \yps{} band are 774 therefore more likely to have a higher false-positive rate than the 775 other stack sources. 776 777 The stack photometry output files consist of a set of FITS table 778 catalogs, with one file for each filter. Within these files, there 779 are multiple table extensions that include: the measurements of 780 sources based on the PSF model; aperture like parameters such as the 781 Petrosian flux and radius; the convolved galaxy model fits; and the 782 radial aperture measurements. \note{is this list complete?} Once the 783 photometry is complete, a row is added to the 784 \ippdbtable{staticskyResult} table with basic statistics from the 785 analysis. 786 787 The stack photometry output catalogs are re-calibrated for both 788 photometry and astrometry in a process very similar to the 789 \ippstage{camera} calibration stage. In the case of this 790 \ippstage{skycal} stage, each skycell is processed independently. 791 Because of this independence, when queued for processing, the entries 792 in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky\_id} 793 and \ippdbcolumn{stack\_id} entries of the parent data directly. As 794 in the \ippstage{camera} stage, the \ippprog{psastro} program reads in 795 the stack photometry catalog, and produces a calibrated output. A 796 different processing recipe is supplied to \ippprog{psastro}, which 797 controls for the different data. The same reference catalog is used 798 for the \ippstage{camera} and \ippstage{stack} calibration stages. 799 Upon completion, the analysis statistics are written to the 800 \ippdbtable{skycalResult} table. \note{Any difference in output formats?} 801 802 \subsection{Forced Warp Photometry} 803 \label{subsec: fullforce} 804 805 Traditionally, projects which use multiple exposures to increase the 806 depth and sensitivity of the observations have generated something 807 equivalent to the \ippstage{stack} images produced by the IPP analysis 808 (c.f, CFHT Legacy survey, COSMOS, etc). In theory, the photometry of 809 the \ippstage{stack} images produces the ``best'' photometry catalog, 810 with best sensitivity and the best data quality at all magnitudes. In 811 practice, these images have some significant limitations due to the 812 difficulty of modelling the PSF variations. This difficulty is 813 particularly severe for the Pan-STARRS $3\pi$ survey stacks due to the 814 combination of the substantial mask fraction of the individual input 815 exposures, the large instrinsic image quality variations within a 816 single exposure, and the wide range of image quality conditions under 817 which data were obtained and used to generate the $3\pi$ PV3 stacks. 818 819 For any specific stack, the point spread function at a particular 820 location is the result of the combination of the point spread 821 functions for those individual exposures which went into the stack at 822 that point. Because of the high mask fraction, the exposures which 823 contributed to pixels at one location may be somewhat different just a 824 few tens of pixels away. In the end, the \ippstage{stack} images have 825 a effective point spread function which is not just variable, but 826 changing significantly on small scales in a highly textured fashion. 827 828 Any measurement which relies on a good knowledge of the PSF at the 829 location of an object either needs to determine the PSF variations 830 present in the \ippstage{stack} image, or the measurement will be 831 somewhat degraded. The highly textured PSF variations make this a 832 very challenging problem: not only would such a PSF model require an 833 unusually fine-grained PSF model, there would likely not be enough PSF 834 stars in a given \ippstage{stack} image to determine the model at the 835 resolution required. The IPP photometry analysis code uses a PSF 836 model with 2D variations using a grid of at most $6\times 6$ samples 837 per skycell, a number reasonably well-matched to the density of stars 838 at most moderate Galactic latitudes. This scale is far too large to 839 track the fine-grained changes apparent in the stack images. 840 841 Thus PSF photometry as well as convolved galaxy models in the stack 842 are degraded by the PSF variations. Aperture-like measurements are in 843 general not as affected by the PSF variations, as long as the aperture 844 in question is large compared to the FWHM of the PSF. 845 846 %% The IPP team initially explored the option of convolving each input 847 %% warp to a single target PSF chosen to match the worst of the input 848 %% images for a given stack. 849 850 The PV3 $3\pi$ analysis solves this problem by using the sources 851 detected in the stack images and performing forced photometry on the 852 individual warp images used to generate the stack. This 853 \ippstage{fullforce} analysis is performed on all warps for a single 854 skycell and filter as a single unit, as this matches the arrangement 855 of the input source catalog from the \ippstage{skycal} stage. When 856 processing is queued for this stage, an entry is added to the 857 \ippdbtable{fullForceRun} primary database table linking to the 858 specific \ippdbcolumn{skycal\_id} entry that will be used as the 859 catalog for the photometry. The \ippdbcolumn{warp\_id} values for the 860 input \ippstage{warp} stage images that contributed to the 861 \ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are 862 then added to the \ippdbtable{fullForceInput} table, linked to the 863 primary table by the \ippdbcolumn{ff\_id} identifier. The individual 864 jobs for each warp are then run, which passes the \ippstage{warp} 865 stage image products along with the \ippstage{skycal} catalog to the 866 \ippprog{psphotFullForce} program. 867 868 In this program, the positions of sources are loaded from the input 869 catalog. PSF stars are pre-identified \note{how?} and a PSF model 870 generated for each \ippstage{warp} image based on those stars, using 871 the same stars for all warps to the extent possible (PSF stars which 872 are excessively masked on a particular image are not used to model the 873 PSF). \note{this doesn't seem correct, as each warp is run 874 independently.} The PSF model is fitted to all of the known source 875 positions in the warp images. Aperture magnitudes, Kron magnitudes, 876 and moments are also measured at this stage for each warp. Note that 877 the flux measurement for a faint, but significant, source from the 878 stack image may be at a low significance (less than the $5\sigma$ 879 criterion used when the photometry is not run in this forced mode) in 880 any individual warp image; the flux may even be negative for specific 881 warps. When combined together, these low-significance measurements 882 will result in a signficant measurement as the signal-to-noise 883 increases by $\sqrt{N}$. 884 885 Upon completion of the forced photometry (for point sources as well as 886 galaxies, discussed below), an entry is added to the 887 \ippdbtable{fullForceResult} table with the processing statistics for 888 that combination of \ippdbcolumn{ff\_id} and \ippdbcolumn{warp\_id}. 889 Once all of the entries in the \ippdbtable{fullForceInput} table have 890 finished, a summary operation is run to generate an appropriate 891 average value for each measurement, by combining the measurements from 892 each of the inputs. The output catalogs listed in the 893 \ippdbtable{fullForceResult} table are passed to the 894 \ippprog{psphotFullForceSummary} to do this averaging. \note{describe 895 what is done} When this completes, an entry is added to the 896 \ippdbtable{fullForceSummary}, and the \ippdbtable{fullForceRun} entry 897 is marked as completed. 898 899 \subsubsection{Forced Galaxy Models} 900 \note{CZW: is this the appropriate place for this section?} 901 902 The convolved galaxy models are also re-measured on the 903 \ippstage{warp} images by the \ippstage{fullforce} stage analysis. In 904 this analysis, the galaxy models determined by the 905 \ippstage{staticsky} photometry analysis are used to seed the analysis 906 in the individual \ippstage{warp} images. The purpose of this 907 analysis is the same as the \ippstage{fullforce} PSF photometry: the 908 PSF of the \ippstage{stack} image is poorly determined due to the 909 masking and PSF variations in the inputs. Without a good PSF model, 910 the PSF-convolved galaxy models are of limited accuracy. 911 912 In the \ippstage{fullforce} galaxy model analysis, we assume that the 913 galaxy position and position angle, along with the Sersic index if 914 appropriate, have been sufficiently well determined in the 915 \ippstage{staticsky} analysis. In this case, the goal is to determine 916 the best values for the major and minor axis of the elliptical contour 917 and at the same time the best normalization corresponding to the best 918 elliptical shape, and thus the best galaxy magnitude value. 919 920 For each \ippstage{warp} image, the \ippstage{staticsky} value for the 921 major and minor axis are used as the center of a $7\times{} 7$ grid 922 search of the major and minor axis parameter values. The grid spacing 923 is defined as a function of the signal-to-noise of the galaxy in the 924 stack image so that bright galaxies are measured with a much finer 925 grid spacing that faint galaxies \note{need to quantify this}. For 926 each grid point, the major and minor axis values at that point are 927 determined for the model. The model is then generated and convolved 928 with the PSF model for the \ippstage{warp} image at that point. The 929 resulting model is then compared to the \ippstage{warp} pixel data 930 values and the best fit normalization value is defined. The 931 normalization and the $\chi^2$ value for each grid point is recorded. 932 933 For a given galaxy, the result is a collection of $\chi^2$ values for 934 each of the grid points spanning all \ippstage{warp} images. A single 935 $\chi^2$ grid can then be made by combining each grid point across the 936 inputs. The combined $\chi^2$ for a single grid point is simply the 937 sum of all $\chi^2$ values at that point. If, for a single \ippstage{warp} 938 image, the galaxy model is excessively masked, then that image will be 939 dropped for all grid points for that galaxy. The reduced $\chi^2$ 940 values can be determined by tracking the total number of pixels 941 used across all inputs to generate the combined $\chi^2$ values. From 942 the combined grid of $\chi^2$ values, the point in the grid with the 943 minimum $\chi^2$ is found. Quadratic interpolation is used to 944 determine the major, minor axis values for the interpolated minimum 945 $\chi^2$ value. The errors on these two parameters is then found by 946 determining the contour at which the \note{reduced?} $\chi^2$ 947 increases by 1. 948 949 Thus the \ippstage{fullforce} galaxy analysis uses the PSF information 950 from each \ippstage{warp} to determine a best set of convovled galaxy 951 models for each object in the \ippstage{skycal} catalog. 952 \note{discuss the subset of galaxy models and objects}. 953 954 \subsection{Difference Images} 955 \label{subsec: diff} 956 Two of the primary science drivers for the Pan-STARRS system are the 957 search hazardous asteroids and the search for Type Ia supernovae to 958 measure the history of the expansion of the universe. Both of these 959 projects require the discovery of faint, transient source in the 960 images. For the hazardous asteroids, and solar system studies in 961 general, the sources are transient because they are moving between 962 observations; supernovae are stationary but transient in brightness. 963 In both cases, the discovery of these sources can be enhanced by 964 subtracting a static reference image from the image taken at a certain 965 epoch. The quality of such a difference image can be enhanced by 966 convolving one or both of the images so that the PSFs in the two 967 images are matched. \note{discuss Alard-Lupton}. 968 969 In the \ippstage{diff} stage, the IPP generates diffferece images for 970 appropriately specified pairs of images. It is possible for the 971 difference image to be generated from a pair of \ippstage{warp} stage 972 images, from a \ippstage{warp} and a \ippstage{stack} of some variety, 973 or from a pair of \ippstage{stack} stage images. During the PS1 974 survey, pairs of exposures, call TTI pairs (see~\note{Survey 975 Strategy}), were obtained for each pointing within a $\approx$ 1 976 hour period in the same filter, and to the extent possible with the 977 same orientation and boresite position. The standard PS1 nightly 978 processing generated difference images from the resulting pairs of 979 \ippstage{warp} images. The nightly processing generated 980 \ippstage{stack} images for the Medium Deep fields, and these were 981 combined with a template reference \ippstage{stack} image to generate 982 ``stack-stack diffs'' each night they were observed. For the PV3 983 $3\pi$ processing, the entire collection of \ippstage{warp} stage 984 images for the survey were combined with images generated by the 985 \ippstage{stack} processing to generate ``warp-stack diffs''. 986 987 When a \ippstage{diff} processing is defined, an entry is added to the 988 \ippdbtable{diffRun} table, and the appropriate input images are added 989 to the \ippdbtable{diffInputSkyfile} table, with one entry for each 990 skycell that are covered by the images. For a \ippstage{diff} 991 generated from two \ippstage{warp} stage products, the input images 992 have their \ippdbcolumn{warp\_id} values recorded in the 993 \ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that 994 overlaps. If two \ippstage{stack} stages are to be used in the 995 difference, their \ippdbcolumn{stack\_id} entries are recorded in the 996 \ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields. As each 997 \ippstage{stack} only covers a single skycell, the \ippstage{diff} is 998 usually defined indirectly, using other information from the 999 \ippdbtable{stackRun} table to select appropriate 1000 \ippdbcolumn{stack\_id} values. Similarly, \ippstage{diff} processing 1001 is defined for the mixed case by creating entries that populate one of 1002 \ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of 1003 \ippdbcolumn{warp2} and \ippdbcolumn{stack2}. In all cases, the 1004 minuend of the subtraction to be performed is the ``1'' entry, and the 1005 subtrahend is the ``2'' entry. 1006 1007 Jobs are created based on the entries of 1008 \ippdbtable{diffInputSkyfile}, with the appropriate images and 1009 catalogs passed to the \ippprog{ppSub} program. This does the 1010 subtraction, as well as the photometry of any sources detected in the 1011 \ippstage{diff} image. The algorithm used for PSF matching is 1012 described in \citet{waters2017}. Upon completion of these jobs, 1013 statistics about the processing are written to an entry in the 1014 \ippdbtable{diffSkyfile} table. An \ippmisc{advance} checks for the 1015 completion of all of the components listed in 1016 \ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun} 1017 entry as such. 1018 1019 \section{IPP Software Subsystems} 1020 \label{sec: subsystems} 1021 1022 The IPP relies on a number of common libraries and programs to handle 1023 various tasks that are shared between multiple stages of the 1024 processing. These subsystems are described in this section, to 1025 provide an introduction to these essential components that underlie 1026 the rest of the pipeline. 262 1027 263 1028 \subsection{Nebulous} … … 929 1694 \note{This likely needs cleaning up and more information.} 930 1695 931 \section{IPP Data Processing Stages}932 \label{sec: stages}933 934 935 \subsection{Summit copy}936 \label{subsec: summit copy}937 938 As exposures are taken by the PS1 telescope \& GPC1 camera system, the939 data from the 60 OTA devices are read out by the camera software940 wsystem and written to disk on a collection of computers at the summit941 in the PS1 facility called ``pixel servers.'' After the images are942 written to disk, a summary listing of the information about the943 exposure and the chip images are added to the summit datastore.944 945 During night-time operations, while the summit datastore is being946 populated, the IPP subsystem called \ippstage{summitcopy} monitors the947 datastores listed in the \ippdbtable{pzDatastore} table of the948 database in order to discover new exposures ready for download. Once949 a new exposure has been listed on the datastore, \ippstage{summitcopy}950 adds an entry of the exposure to a table in the processing database951 (\ippdbtable{summitExp}), indexed by an identifier that simply952 increments the number of exposures announced by the summit, the953 \ippdbcolumn{summit\_id}. This tells the \ippstage{summitcopy} system954 to look for the list of chips, which are then added to another table955 (\ippdbtable{summitImfile}). This system then attempts to download956 the chips (registering the results of those operations into the957 \ippdbtable{pzDownloadExp} and \ippdbtable{pzDownloadImfile} tables)958 from the summit pixel servers via an http request. As the image files959 are downloaded, their MD5 checksum values are calculated and compared960 with the value reported by the summit datastore. Download failures961 are rare and marked with a non-zero \ippdbcolumn{fault}, allowing for962 a manual recovery, rather than automatically rejecting the failed963 chips. Once all the components of the exposure have been downloaded,964 they are further entered into the \ippdbtable{newExp} and965 \ippdbtable{newImfile} tables, which index the exposures by966 \ippdbcolumn{exp\_id}. This switch in index indicates that the967 exposure has successfully been copied from the summit to the IPP968 cluster, and that further processing is no longer dependent on outside969 resources.970 971 \subsection{Image Registration}972 \label{subsec: registration}973 974 Once the chips for an exposure have all been downloaded, the exposure975 is ready to be registered. In this context, `registration' refers to976 the process of adding them to the database listing of known, raw977 exposures (not to be confused with 'registration' in the sense of978 pixel re-alignment). The result of the registration analysis is an979 entry for each exposure in the \ippdbtable{rawExp} table, and one for980 each chip in the \ippdbtable{rawImfile} table. These tables are981 critical for downstream processing to identify what exposures are982 available for processing in any other stage. At the registration983 stage, a large amount of descriptive metadata for each chip is added984 to the \ippdbtable{rawImfile} table, the majority of which is985 extracted from the chip FITS file headers (e.g., RA, DEC, FILTER) and986 some of which is determined by a quick analysis of the pixels (e.g.,987 mean pixel values, standard deviation). The chip-level information is988 merged into a set of exposure-level metadata and added to the989 \ippdbtable{rawExp} table entry. The exposure-level metadata may be990 the same as any one of the chip, in a case where the values are991 duplicated across the chip files (e.g., the name of the telescope or992 the date \& time of the exposure), or it may be a calculation based on993 the values from each chip (e.g., average of the average pixel values).994 995 Unlike much of the rest of the IPP stage, the raw exposures may only996 have a single entry in the registration tables of the processing997 database tables (\ippdbtable{rawExp} and \ippdbtable{rawImfile}).998 999 For GPC1, the image registration stage is also the stage at which the1000 \ippprog{burntool} analysis is run. This analysis is more completely1001 described in \citet{waters2017}. In brief, the \ippprog{burntool}1002 program identifies bright sources on the image, and identifies1003 persistence trails that result from the incomplete transfer of charge.1004 As this charge can leak out in subsequent exposures, the burntool1005 analysis is run sequentially on the exposures, based on the1006 observation date and time listed in the headers, with the results1007 stored in an text table. As a result of the sequential nature of this1008 analysis, the registration of exposures is blocked until the1009 \ippprog{burntool} has been run on the previous exposures.1010 1011 Once the registration process has finished, new science exposures that1012 have an \ippdbcolumn{obs\_mode} value that indicates they are part of1013 a particular science survey are automatically launched into the1014 science analysis by defining entries for the \ippstage{chip}1015 processing stage, as described above. This analysis can be relaunched1016 multiple times, such as for the large scale PV3 reprocessing.1017 However, this automatic process ensures the shortest time between1018 observation and analysis, which is particularly important in the1019 search for transient sources.1020 1021 \subsection{Chip Processing}1022 \label{subsec: chip}1023 1024 The science analysis of an exposure begins with the \ippstage{chip}1025 stage, which operates on the individual OTA image files. This1026 analysis step has two main goals: detrending the image to remove the1027 instrumental signature from the pixel values, and the detection of1028 astronomical sources in the objects. Based on the entry the1029 \ippdbtable{chipRun} primary table defining the processing details1030 (with the \ippdbcolumn{state} column indicating it needs processing),1031 and the associated information listed in the \ippdbtable{rawImfile},1032 jobs can be spawned for each component OTA. The \ippprog{pantasks}1033 environment managing the jobs attempts to target the processing host1034 to one that should host the OTA, to reduce number of operations done1035 on remote data. In practice, this targeted processing has not had as1036 large of an effect as was originally intended, as the data volume has1037 reduced the ability of any one node to reliably contain a particular1038 OTA. The targeted processing has probably reduced the network load1039 somewhat but it has not been as critical of a requirement as1040 originally expected.1041 1042 %% In the \ippstage{chip} stage,1043 %% the individual OTA image files are processed independently in parallel1044 %% within the data processing cluster. \note{move this to kihei1045 %% discussion?} Within the processing computer cluster, most of the1046 %% data storage resources are in the form of computers with large raids1047 %% as well as substantial processing capability. The processing system1048 %% attempts to locate one copy of specific raw registered data on1049 %% pre-defined computers that have been set as storage targets for that1050 %% OTA. The processing system is aware of this data localization and1051 %% attempts to target the processing for each OTA to the machine on which1052 %% the data for that detector is stored. The output products are then1053 %% primarily saved back to the same machine. This `targetted' processing1054 %% was an early design choice to minimize the system wide network load1055 %% during processing. In practice, as computer disks filled up at1056 %% different rates, the data has not been localized to a very high1057 %% degree.1058 1059 The actual image processing is performed by the \ippprog{ppImage}1060 program. This program reads the raw data into memory and applies the1061 detrend corrections \citep[see][]{waters2017} to each cell in the OTA1062 (which are stored as different extensions in the FITS file format),1063 and then mosaics the cells into a single contiguous \ippstage{chip}1064 stage image. This step also creates in memory additional images to1065 hold the mask data, which indicates which pixels may not be valid, and1066 the variance image, constructed as the Poissonian noise on the number1067 of electrons detected based on the original pixel value and the1068 detector gain. A background model is then fit across the image and1069 subtracted to remove the expected contribution from the sky1070 \citep[see][]{waters2017} for details.1071 1072 With the image calibration procedure finished, object identification1073 and photometry can be performed. Although this can be done using a1074 stand alone program, \ippprog{psphot}, the underlying functions are1075 contained in a library that allows \ippprog{ppImage} to directly do1076 this analysis, removing the need to write out and re-read the image1077 data. The details of the detection and characterization of the1078 sources in the image are provided in \citet{magnier2017b}.1079 1080 The results of the image processing are then written to disk,1081 including the science, mask, and variance images, the background model1082 subtracted, the PSF model used in the photometry process, and a FITS1083 catalog of detected sources. Additional binned images of the full OTA1084 are also saved, providing $16\times{}16$ and $256\times{}256$ pixel1085 binning scales for quick visualization. The processing log and a1086 selection of summary metadata describing the processing results are1087 also written to disk. This metadata is used to populate a row in the1088 \ippdbtable{chipProcessedImfile} table (linked to the1089 \ippdbtable{chipRun} entry by a shared \ippdbcolumn{chip\_id} value)1090 to indicate that the processing of this OTA is complete.1091 1092 As each OTA is processed independently of the others across a number1093 of computers, the \ippprog{pantasks} managing the jobs periodically1094 runs an \ippmisc{advance} task that checks that the number of rows in1095 \ippdbtable{chipProcessedImfile} with \ippdbcolumn{fault} equal to1096 zero matches the associated number of rows in \ippdbtable{rawImfile}.1097 If this condition is met, than all processing for that exposure is1098 finished, and the \ippdbcolumn{state} field is set to ``full''. If1099 the \ippdbtable{chipRun}.\ippdbcolumn{end\_stage} field is set to1100 \ippstage{chip}, then no further action is taken. However, this field1101 is usually set to a subsequent stage (most often \ippstage{warp}),1102 then an entry for this exposure is added to the \ippdbtable{camRun}1103 table, and processing continues.1104 1105 %% The \ippstage{chip} processing stage consists of: reading the raw image into1106 %% memory, applying the detrending steps \citep[see][]{waters2017},1107 %% stiching the individual OTA cells into a single chip image, detection1108 %% and characterization of the sources in the image1109 %% \citep[see][]{magnier2017b}, and output of the various data products.1110 %% These include the detrended chip image, variance image, and mask1111 %% image, as well as the FITS catalog of detected sources. The PSF model1112 %% and background model are also saved, along with a processing log. A1113 %% selection of summary metadata describing the processing results are1114 %% saved and written to the processing database along with the completion1115 %% status of the process. Finally, binned chip images are generated (on1116 %% two scales, binned by 16 and 256 pixels) for use in the visualization1117 %% system of the processing monitor tool. \note{describe elsewhere?}1118 1119 %% The database structure for the \stage{chip} stage mimics that of raw1120 %% data, with a \ippdbtable{chipRun} characterizing the processing of a1121 %% single exposure, mapping to a set of \ippdbtable{chipProcessedImfile}1122 %% entries for each OTA via a common \ippdbcolumn{chip\_id}.1123 1124 \subsection{Camera Calibration}1125 \label{subsec: camera}1126 1127 After sources have been detected and measured for each of the chips,1128 the next stage is to perform a basic calibration of the full exposure1129 in the \ippstage{camera} stage. This runs as a single job for the1130 entire exposure, passing the collection of FITS table catalogs1131 generated from each OTA in the \ippstage{chip} stage to the1132 \ippprog{psastro} program. Although the full catalog is loaded, the1133 calibration primarily concerns the positions ($x_{\rm ccd}, y_{\rm1134 ccd}$) and the instrumental PSF magnitudes. The header information1135 in these catalogs is used to determine the coordinates of the1136 telescope boresite (RA, DEC, position angle). These three coordinates1137 are used, along with a pre-determined model of the OTA layout within1138 the camera, to generate an initial guess for the astrometry of each1139 chip. Reference star coordinates and magnitudes are loaded from a1140 reference catalog for a region corresponding to the boundaries of the1141 exposure, padded by a large fraction (25\%) of the exposure diameter1142 to help guarantee a solution in the case of a modest pointing error.1143 The guess astrometry is used to match the reference catalog to the1144 observed stellar positions in the focal plane coordinate system. Once1145 an acceptable match is found, the astrometric calibration of the1146 individual chips is performed, including a fit to a single model for1147 the distortion introduced by the camera optics. After the astrometic1148 analysis is completed, the photometric calibration is determined using1149 the final match to the reference catalog. At this stage,1150 pre-determined color terms may be included to convert the reference1151 photometry to an appropriate photometric system. For PS1, this is1152 used to generate synthetic w-band photometry for areas where no1153 PS1-based calibrated w-band photometry is available. For more1154 details, see \cite{magnier2017c}. The result of these calibrations is1155 stored as a single multi-extension FITS table containing the results1156 from each OTA as a separate extension.1157 1158 In addition to the astrometric and photometric calibrations, the1159 \ippstage{camera} stage also generates the dynamic masks for the1160 images. These include masking for optical ghosts, glints, saturated1161 stars, diffraction spikes, and electronic crosstalk. The mask images1162 generated by the \ippstage{chip} stage are updated with these dynamic1163 masks and a new set of files are saved for the downstream analysis1164 stages. The \ippstage{camera} stage also merges the binned chip1165 images (see~\ref{sec:chip}) into single jpeg images of the entire1166 focal plane. These jpeg images can then be displayed by the process1167 monitoring system to visualize the data processing.1168 1169 Again, summary metadata is saved to disk as well, and the results1170 listed therein are used to populate a row in the1171 \ippdbtable{camProcessedExp} database table. As the full exposure is1172 processed all at once, this update also updates the associated1173 \ippdbtable{camRun} entry, linked by the \ippdbcolumn{cam\_id}. As1174 with the \ippstage{chip} stage, the1175 \ippdbtable{camRun}.\ippdbcolumn{end\_stage} is for a subsequent1176 stage, an appropriate entry is added to the \ippdbtable{fakeRun}1177 table.1178 1179 \subsection{Fake Analysis}1180 \label{subsec: fake}1181 1182 The \ippstage{fake} stage was originally designed to do false source1183 injection and recovery, in order to determine the detection efficiency1184 of sources on the exposure. However, early in the design of the IPP,1185 this task was moved to the rest of the photometry analysis done at the1186 \ippstage{chip} stage. Removing the stage would require significant1187 changes to the database schema. As a result, this conveniently named1188 stage generally does no actual data processing, and consists mainly of1189 database operations to move the exposure on to the \ippstage{warp}1190 stage. The operations mimic the \ippstage{chip} stage, with1191 individual jobs run for each OTA that update rows in the1192 \ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that1193 updates the \ippdbtable{fakeRun} table and promotes the exposure to1194 the next stage by adding a row to the \ippdbtable{warpRun} table.1195 1196 \subsection{Image Warping}1197 \label{subsec: warp}1198 1199 The \ippstage{warp} stage moves the data from a given exposure beyond1200 away from being camera specific and towards a uniform sky oriented1201 arrangement. There are a number of ``tessellations'' defined and used1202 by the IPP to define the extent and scaling of images on this uniform1203 arrangement. A tessellation can be defined for a limited region, such1204 as M31 or other fields of particular interest that can be well1205 described by a single tangent plane projection, or for larger regions1206 which have multiple projection centers. For the $3\Pi$ survey, the1207 \ippmisc{RINGS.V3} tessellation was used that used projection centers1208 spaced every four degrees in both RA and DEC, with $0\farcs{}25$1209 pixels. These projections are further broken down into ``skycells''1210 that form a $10\times{}10$ grid within the projection, with an overlap1211 region of 60" between adjacent skycells to ensure that objects are not1212 split on all images.1213 1214 These tessellations are stored in the DVO format, with1215 \ippdbtable{SkyTable} entries defining the projection centers and1216 image boundaries for all the skycells. The first step of the1217 \ippstage{warp} stage is determining which skycells overlap with the1218 input exposure. These overlaps are determined by the1219 \ippprog{dvoImageOverlaps} program, which compares the astrometrically1220 calibrated catalog from the \ippstage{camera} stage to the1221 \ippdbtable{SkyTable} entries. The output of this command is used to1222 populate the \ippdbtable{warpSkyCellMap} table in the database, which1223 contains a row for each skycell and OTA that overlap. This results in1224 more rows than there are OTAs, as each skycell can contain1225 contributions from multiple OTAs.1226 1227 Once this mapping has been defined, jobs to construct each skycell are1228 run, passing the \ippstage{camera} stage catalog and the1229 \ippstage{chip} stage images (including the variance images and the1230 updated masks) to the \ippprog{pswarp} program. For details on the1231 warping algorithm, see \cite{waters2017}. The output of this program1232 are the geometrically transformed images containing all input pixels1233 warped to the common skycell pixel grid, which can subsequently be1234 used for stacking and difference image analysis. The image, mask, and1235 variance generated at this stage will be available from the image1236 extraction tools at the MAST archive at STScI as part of the DR2 data1237 release. A catalog is also generated containing the locations of1238 sources from the input catalog that fall within area of the1239 \ippstage{warp}.1240 1241 When the jobs have completed, an entry for the skycell is added to the1242 \ippdbtable{warpSkyfile} database table, linked to the1243 \ippdbtable{warpRun} entry by a common \ippdbcolumn{warp\_id}. An1244 \ippmisc{advance} task again checks that all potential skycells have1245 been generated. At this point, the direct promotion of exposures from1246 one stage to the next stops, as the logic for matching exposures for1247 combination is more complicated than simply adding a single entry (as1248 discussed above).1249 1250 \subsection{Stack Combination}1251 \label{subsec: stack}1252 1253 The skycell images generated by the \ippstage{warp} process are added1254 together to make deeper, higher signal-to-noise images in the1255 \ippstage{stack} stage. These stacked images also fill in coverage1256 gaps between different exposures, resulting in an image of the sky1257 with more uniform coverage than a single exposure.1258 1259 In the IPP processing, stacks may be made with various options for the1260 input images. During nightly science processing, the 8 exposures per1261 filter for each Medium Deep field are combined into a set of stacks1262 for that field. These so-called `nightly stacks' are used by the1263 transient survey projects to detect faint supernovae, among other1264 transient events. For the PV3 $3\pi$ analysis, all images in each1265 filter from the observations for this survey were stacked together to1266 generate a single set of images with $\sim 10 - 20\times$ the exposure1267 of the individual survey exposures.1268 1269 For the PV3 processing of the Medium Deep fields, stacks have been1270 generated for the nightly groups and for the full depth using all1271 exposures, producing ``deep stacks''. In addition, a 'best seeing'1272 set of stacks have been produced \note{using image quality cuts to be1273 described}. We have also generated out-of-season stacks for the1274 Medium Deep fields, in which all image not from a particular observing1275 season for a field are combined into a stack. These later stacks are1276 useful as deep templates when studying long-term transient events in1277 the Medium Deep fields as they are not (or less) contaminated by the1278 flux of the transients from a given season.1279 1280 When a given set of \ippstage{stack} stage are defined, exposures with1281 existing \ippstage{warp} entries that match the filter, position, and1282 other criteria such as seeing are grouped by their skycell. An entry1283 is then added for each skycell in the \ippdbtable{stackRun} table,1284 with the \ippdbcolumn{warp\_id} entries for the exposures added to the1285 \ippdbtable{stackInputSkyfile} table, linked to the1286 \ippdbtable{stackRun} entry by the \ippdbcolumn{stack\_id} field.1287 This defines the mapping for which exposures contribute to the1288 \ippstage{stack}. This breaks exposures into single skycells, but as1289 adjacent \ippstage{stack} skycells may contain inputs from different1290 exposures, there is no simple way to group the processing at the1291 \ippstage{stack} stage into exposures.1292 1293 The \ippstage{stack} jobs pass the information about the input images1294 and catalogs to the \ippprog{ppStack} program, which performs the1295 image combinations. See~\cite{waters2017} for details on the stack1296 combination algorithm. In addition to the standard image, mask, and1297 variance produced at other stage, additional images are constructed1298 with information about the contributions to each pixel. A number1299 image contains the number of input exposures used for each pixel,1300 along with an exposure time map, and a weighted exposure time map that1301 scales the exposure time based on the relative variance of each input.1302 These images for the $3\Pi$ analysis are currently available from the1303 MAST image extraction tools at STSci.1304 1305 Upon completing the generation of these images, a row is added to the1306 \ippdbtable{stackSumSkyfile} table with statistics about1307 \ippstage{stack} processing. As this completes all processing for the1308 entry, no \ippmisc{advance} job is required.1309 1310 \subsection{Stack Photometry}1311 \label{subsec: staticsky}1312 1313 Although images are generated in the \ippstage{stack} stage of the1314 IPP, the source detection and extraction analysis of those images is1315 deferred to the \ippstage{staticsky} stage. This separation is1316 maintained because the photometry analysis of the \ippstage{stack}1317 images is performed on all 5 filters simultaneously. By deferring1318 this analysis, the processing system may also decouple the generation1319 of the pixels from the source detection. This makes the sequencing of1320 analysis somewhat easier and less subject to blocks due to a failure1321 in the stacking analysis. Similar to the \ippstage{stack} stage, an1322 entry is created in the \ippdbtable{staticskyRun} table, linked to a1323 series of rows in the \ippdbtable{staticskyInput} table by a common1324 \ippdbcolumn{sky\_id}, each of which also contains the appropriate1325 \ippdbcolumn{stack\_id} entries for the skycell under consideration.1326 1327 The input images are passed to the \ippprog{psphotStack} program,1328 which does the analysis. The stack photometry algorithms are1329 described in detail in \cite{magnier2017b}. In short, sources are1330 detected in all 5 filter images down to the $5\sigma$ significance.1331 The collection of detected sources is merged into a single master1332 list. If a source is detected in at least two bands, or only in1333 \yps{} band, then a PSF model is fitted to the pixels of the other1334 bands in which the source was not detected. This forced photometry1335 results in lower significance measurements of the flux at the1336 positions of objects which are thought to be real sources, by virtue1337 of triggering a detection in at least two bands. The relaxed limit1338 for \yps{} band is included to allow for searches of \yps{} dropout1339 objects: it is known that faint, high-redshift quasars may be detected1340 in \yps{} band only. Sources detected only in \yps{} band are1341 therefore more likely to have a higher false-positive rate than the1342 other stack sources.1343 1344 The stack photometry output files consist of a set of FITS table1345 catalogs, with one file for each filter. Within these files, there1346 are multiple table extensions that include: the measurements of1347 sources based on the PSF model; aperture like parameters such as the1348 Petrosian flux and radius; the convolved galaxy model fits; and the1349 radial aperture measurements. \note{is this list complete?} Once the1350 photometry is complete, a row is added to the1351 \ippdbtable{staticskyResult} table with basic statistics from the1352 analysis.1353 1354 The stack photometry output catalogs are re-calibrated for both1355 photometry and astrometry in a process very similar to the1356 \ippstage{camera} calibration stage. In the case of this1357 \ippstage{skycal} stage, each skycell is processed independently.1358 Because of this independence, when queued for processing, the entries1359 in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky\_id}1360 and \ippdbcolumn{stack\_id} entries of the parent data directly. As1361 in the \ippstage{camera} stage, the \ippprog{psastro} program reads in1362 the stack photometry catalog, and produces a calibrated output. A1363 different processing recipe is supplied to \ippprog{psastro}, which1364 controls for the different data. The same reference catalog is used1365 for the \ippstage{camera} and \ippstage{stack} calibration stages.1366 Upon completion, the analysis statistics are written to the1367 \ippdbtable{skycalResult} table. \note{Any difference in output formats?}1368 1369 \subsection{Forced Warp Photometry}1370 \label{subsec: fullforce}1371 1372 Traditionally, projects which use multiple exposures to increase the1373 depth and sensitivity of the observations have generated something1374 equivalent to the \ippstage{stack} images produced by the IPP analysis1375 (c.f, CFHT Legacy survey, COSMOS, etc). In theory, the photometry of1376 the \ippstage{stack} images produces the ``best'' photometry catalog,1377 with best sensitivity and the best data quality at all magnitudes. In1378 practice, these images have some significant limitations due to the1379 difficulty of modelling the PSF variations. This difficulty is1380 particularly severe for the Pan-STARRS $3\pi$ survey stacks due to the1381 combination of the substantial mask fraction of the individual input1382 exposures, the large instrinsic image quality variations within a1383 single exposure, and the wide range of image quality conditions under1384 which data were obtained and used to generate the $3\pi$ PV3 stacks.1385 1386 For any specific stack, the point spread function at a particular1387 location is the result of the combination of the point spread1388 functions for those individual exposures which went into the stack at1389 that point. Because of the high mask fraction, the exposures which1390 contributed to pixels at one location may be somewhat different just a1391 few tens of pixels away. In the end, the \ippstage{stack} images have1392 a effective point spread function which is not just variable, but1393 changing significantly on small scales in a highly textured fashion.1394 1395 Any measurement which relies on a good knowledge of the PSF at the1396 location of an object either needs to determine the PSF variations1397 present in the \ippstage{stack} image, or the measurement will be1398 somewhat degraded. The highly textured PSF variations make this a1399 very challenging problem: not only would such a PSF model require an1400 unusually fine-grained PSF model, there would likely not be enough PSF1401 stars in a given \ippstage{stack} image to determine the model at the1402 resolution required. The IPP photometry analysis code uses a PSF1403 model with 2D variations using a grid of at most $6\times 6$ samples1404 per skycell, a number reasonably well-matched to the density of stars1405 at most moderate Galactic latitudes. This scale is far too large to1406 track the fine-grained changes apparent in the stack images.1407 1408 Thus PSF photometry as well as convolved galaxy models in the stack1409 are degraded by the PSF variations. Aperture-like measurements are in1410 general not as affected by the PSF variations, as long as the aperture1411 in question is large compared to the FWHM of the PSF.1412 1413 %% The IPP team initially explored the option of convolving each input1414 %% warp to a single target PSF chosen to match the worst of the input1415 %% images for a given stack.1416 1417 The PV3 $3\pi$ analysis solves this problem by using the sources1418 detected in the stack images and performing forced photometry on the1419 individual warp images used to generate the stack. This1420 \ippstage{fullforce} analysis is performed on all warps for a single1421 skycell and filter as a single unit, as this matches the arrangement1422 of the input source catalog from the \ippstage{skycal} stage. When1423 processing is queued for this stage, an entry is added to the1424 \ippdbtable{fullForceRun} primary database table linking to the1425 specific \ippdbcolumn{skycal\_id} entry that will be used as the1426 catalog for the photometry. The \ippdbcolumn{warp\_id} values for the1427 input \ippstage{warp} stage images that contributed to the1428 \ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are1429 then added to the \ippdbtable{fullForceInput} table, linked to the1430 primary table by the \ippdbcolumn{ff\_id} identifier. The individual1431 jobs for each warp are then run, which passes the \ippstage{warp}1432 stage image products along with the \ippstage{skycal} catalog to the1433 \ippprog{psphotFullForce} program.1434 1435 In this program, the positions of sources are loaded from the input1436 catalog. PSF stars are pre-identified \note{how?} and a PSF model1437 generated for each \ippstage{warp} image based on those stars, using1438 the same stars for all warps to the extent possible (PSF stars which1439 are excessively masked on a particular image are not used to model the1440 PSF). \note{this doesn't seem correct, as each warp is run1441 independently.} The PSF model is fitted to all of the known source1442 positions in the warp images. Aperture magnitudes, Kron magnitudes,1443 and moments are also measured at this stage for each warp. Note that1444 the flux measurement for a faint, but significant, source from the1445 stack image may be at a low significance (less than the $5\sigma$1446 criterion used when the photometry is not run in this forced mode) in1447 any individual warp image; the flux may even be negative for specific1448 warps. When combined together, these low-significance measurements1449 will result in a signficant measurement as the signal-to-noise1450 increases by $\sqrt{N}$.1451 1452 Upon completion of the forced photometry (for point sources as well as1453 galaxies, discussed below), an entry is added to the1454 \ippdbtable{fullForceResult} table with the processing statistics for1455 that combination of \ippdbcolumn{ff\_id} and \ippdbcolumn{warp\_id}.1456 Once all of the entries in the \ippdbtable{fullForceInput} table have1457 finished, a summary operation is run to generate an appropriate1458 average value for each measurement, by combining the measurements from1459 each of the inputs. The output catalogs listed in the1460 \ippdbtable{fullForceResult} table are passed to the1461 \ippprog{psphotFullForceSummary} to do this averaging. \note{describe1462 what is done} When this completes, an entry is added to the1463 \ippdbtable{fullForceSummary}, and the \ippdbtable{fullForceRun} entry1464 is marked as completed.1465 1466 \subsubsection{Forced Galaxy Models}1467 \note{CZW: is this the appropriate place for this section?}1468 1469 The convolved galaxy models are also re-measured on the1470 \ippstage{warp} images by the \ippstage{fullforce} stage analysis. In1471 this analysis, the galaxy models determined by the1472 \ippstage{staticsky} photometry analysis are used to seed the analysis1473 in the individual \ippstage{warp} images. The purpose of this1474 analysis is the same as the \ippstage{fullforce} PSF photometry: the1475 PSF of the \ippstage{stack} image is poorly determined due to the1476 masking and PSF variations in the inputs. Without a good PSF model,1477 the PSF-convolved galaxy models are of limited accuracy.1478 1479 In the \ippstage{fullforce} galaxy model analysis, we assume that the1480 galaxy position and position angle, along with the Sersic index if1481 appropriate, have been sufficiently well determined in the1482 \ippstage{staticsky} analysis. In this case, the goal is to determine1483 the best values for the major and minor axis of the elliptical contour1484 and at the same time the best normalization corresponding to the best1485 elliptical shape, and thus the best galaxy magnitude value.1486 1487 For each \ippstage{warp} image, the \ippstage{staticsky} value for the1488 major and minor axis are used as the center of a $7\times{} 7$ grid1489 search of the major and minor axis parameter values. The grid spacing1490 is defined as a function of the signal-to-noise of the galaxy in the1491 stack image so that bright galaxies are measured with a much finer1492 grid spacing that faint galaxies \note{need to quantify this}. For1493 each grid point, the major and minor axis values at that point are1494 determined for the model. The model is then generated and convolved1495 with the PSF model for the \ippstage{warp} image at that point. The1496 resulting model is then compared to the \ippstage{warp} pixel data1497 values and the best fit normalization value is defined. The1498 normalization and the $\chi^2$ value for each grid point is recorded.1499 1500 For a given galaxy, the result is a collection of $\chi^2$ values for1501 each of the grid points spanning all \ippstage{warp} images. A single1502 $\chi^2$ grid can then be made by combining each grid point across the1503 inputs. The combined $\chi^2$ for a single grid point is simply the1504 sum of all $\chi^2$ values at that point. If, for a single \ippstage{warp}1505 image, the galaxy model is excessively masked, then that image will be1506 dropped for all grid points for that galaxy. The reduced $\chi^2$1507 values can be determined by tracking the total number of pixels1508 used across all inputs to generate the combined $\chi^2$ values. From1509 the combined grid of $\chi^2$ values, the point in the grid with the1510 minimum $\chi^2$ is found. Quadratic interpolation is used to1511 determine the major, minor axis values for the interpolated minimum1512 $\chi^2$ value. The errors on these two parameters is then found by1513 determining the contour at which the \note{reduced?} $\chi^2$1514 increases by 1.1515 1516 Thus the \ippstage{fullforce} galaxy analysis uses the PSF information1517 from each \ippstage{warp} to determine a best set of convovled galaxy1518 models for each object in the \ippstage{skycal} catalog.1519 \note{discuss the subset of galaxy models and objects}.1520 1521 \subsection{Difference Images}1522 \label{subsec: diff}1523 Two of the primary science drivers for the Pan-STARRS system are the1524 search hazardous asteroids and the search for Type Ia supernovae to1525 measure the history of the expansion of the universe. Both of these1526 projects require the discovery of faint, transient source in the1527 images. For the hazardous asteroids, and solar system studies in1528 general, the sources are transient because they are moving between1529 observations; supernovae are stationary but transient in brightness.1530 In both cases, the discovery of these sources can be enhanced by1531 subtracting a static reference image from the image taken at a certain1532 epoch. The quality of such a difference image can be enhanced by1533 convolving one or both of the images so that the PSFs in the two1534 images are matched. \note{discuss Alard-Lupton}.1535 1536 In the \ippstage{diff} stage, the IPP generates diffferece images for1537 appropriately specified pairs of images. It is possible for the1538 difference image to be generated from a pair of \ippstage{warp} stage1539 images, from a \ippstage{warp} and a \ippstage{stack} of some variety,1540 or from a pair of \ippstage{stack} stage images. During the PS11541 survey, pairs of exposures, call TTI pairs (see~\note{Survey1542 Strategy}), were obtained for each pointing within a $\approx$ 11543 hour period in the same filter, and to the extent possible with the1544 same orientation and boresite position. The standard PS1 nightly1545 processing generated difference images from the resulting pairs of1546 \ippstage{warp} images. The nightly processing generated1547 \ippstage{stack} images for the Medium Deep fields, and these were1548 combined with a template reference \ippstage{stack} image to generate1549 ``stack-stack diffs'' each night they were observed. For the PV31550 $3\pi$ processing, the entire collection of \ippstage{warp} stage1551 images for the survey were combined with images generated by the1552 \ippstage{stack} processing to generate ``warp-stack diffs''.1553 1554 When a \ippstage{diff} processing is defined, an entry is added to the1555 \ippdbtable{diffRun} table, and the appropriate input images are added1556 to the \ippdbtable{diffInputSkyfile} table, with one entry for each1557 skycell that are covered by the images. For a \ippstage{diff}1558 generated from two \ippstage{warp} stage products, the input images1559 have their \ippdbcolumn{warp\_id} values recorded in the1560 \ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that1561 overlaps. If two \ippstage{stack} stages are to be used in the1562 difference, their \ippdbcolumn{stack\_id} entries are recorded in the1563 \ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields. As each1564 \ippstage{stack} only covers a single skycell, the \ippstage{diff} is1565 usually defined indirectly, using other information from the1566 \ippdbtable{stackRun} table to select appropriate1567 \ippdbcolumn{stack\_id} values. Similarly, \ippstage{diff} processing1568 is defined for the mixed case by creating entries that populate one of1569 \ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of1570 \ippdbcolumn{warp2} and \ippdbcolumn{stack2}. In all cases, the1571 minuend of the subtraction to be performed is the ``1'' entry, and the1572 subtrahend is the ``2'' entry.1573 1574 Jobs are created based on the entries of1575 \ippdbtable{diffInputSkyfile}, with the appropriate images and1576 catalogs passed to the \ippprog{ppSub} program. This does the1577 subtraction, as well as the photometry of any sources detected in the1578 \ippstage{diff} image. The algorithm used for PSF matching is1579 described in \citet{waters2017}. Upon completion of these jobs,1580 statistics about the processing are written to an entry in the1581 \ippdbtable{diffSkyfile} table. An \ippmisc{advance} checks for the1582 completion of all of the components listed in1583 \ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun}1584 entry as such.1585 1586 1696 \subsection{Addstar : DVO Ingest} 1587 1697 \label{subsec: addstar}
Note:
See TracChangeset
for help on using the changeset viewer.
