Changeset 40021
- Timestamp:
- Apr 28, 2017, 4:51:12 AM (9 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/release.2015/ps1.datasystem/datasystem.tex
r40020 r40021 311 311 stage is completed, summary information about the stage is written 312 312 back to the database. In this way, the database records this history 313 of the processing, and also provides the information needed to313 of the processing, and also provides the information needed by 314 314 successive processing stages to begin their own tasks. 315 315 … … 319 319 This same database engine also has instances for other cameras 320 320 processed by the IPP, e.g., GPC2, the test cameras TC1, TC3, and the 321 Imaging Sky Probe (ISP). 321 Imaging Sky Probe (ISP). In general, processing information for 322 different cameras is separate in differnt processing database; merging 323 of output products takes place in DVO. 322 324 323 325 Within the processing database, the various processing stages are … … 371 373 occurs, the system will not process an exposure through subsequent 372 374 stages without the component that has failed temporarily. Since many 373 of the \ippdbcolumn{fault}s which occur are ephemeral, the processing 374 stages are set up to occasional clear and re-try the faulted entries. 375 Thus, automatic processing is able to keep the data flowing even in 376 the face of occasional network glitches or hardware crashes. 375 of the \ippdbcolumn{fault}s which occur are ephemeral due to current 376 conditions of the processing cluster, the processing stages are set up 377 to occasionally clear and re-try the faulted entries. Some faults 378 represent software bugs and in the early stages of processing were 379 accumulated until the corresponding software issue could be addressed; 380 since the start of the PS1 Science Consortium Surveys, these types of 381 faults have largely been eliminated. Thus, automatic processing is 382 able to keep the data flowing even in the face of occasional network 383 glitches or hardware crashes. 384 385 \note{start of section needed a re-read} 377 386 378 387 \subsection{Summit copy} … … 641 650 \label{sec:warp} 642 651 652 \note{need to describe the RINGS.V3 tessellation and others} 653 643 654 The \ippstage{warp} stage moves the data from a given exposure beyond 644 655 away from being camera specific and towards a uniform sky oriented … … 752 763 entry, no \ippmisc{advance} job is required. 753 764 765 \note{end of section needed a re-read} 766 754 767 \subsection{Stack Photometry} 755 768 \label{sec:staticsky} … … 759 772 deferred to the \ippstage{staticsky} stage. This separation is 760 773 maintained because the photometry analysis of the \ippstage{stack} 761 images is performed on all 5 filters simultaneously. By deferring 762 this analysis, the processing system may also decouple the generation 763 of the pixels from the source detection. This makes the sequencing of 764 analysis somewhat easier and less subject to blocks due to a failure 765 in the stacking analysis. Similar to the \ippstage{stack} stage, an 766 entry is created in the \ippdbtable{staticskyRun} table, linked to a 767 series of rows in the \ippdbtable{staticskyInput} table by a common 768 \ippdbcolumn{sky\_id}, each of which also contains the appropriate 769 \ippdbcolumn{stack\_id} entries for the skycell under consideration. 774 images, including convolved galaxy model fitting, is performed on all 775 5 filters simultaneously. By deferring this analysis, the processing 776 system may also decouple the generation of the pixels from the source 777 detection. This makes the sequencing of analysis somewhat easier and 778 less subject to blocks due to a failure in the stacking analysis. 779 Similar to the \ippstage{stack} stage, an entry is created in the 780 \ippdbtable{staticskyRun} table, linked to a series of rows in the 781 \ippdbtable{staticskyInput} table by a common \ippdbcolumn{sky\_id}, 782 each of which also contains the appropriate \ippdbcolumn{stack\_id} 783 entries for the skycell under consideration. 770 784 771 785 The input images are passed to the \ippprog{psphotStack} program, … … 840 854 841 855 Any measurement which relies on a good knowledge of the PSF at the 842 location of an object either needs to determine the PSF variations 843 present in the \ippstage{stack} image, or the measurement will be 844 somewhat degraded. The highly textured PSF variations make this a 845 very challenging problem: not only would such a PSF model require an 846 unusually fine-grained PSF model, there would likely not be enough PSF 847 stars in a given \ippstage{stack} image to determine the model at the 848 resolution required. The IPP photometry analysis code uses a PSF 849 model with 2D variations using a grid of at most $6\times 6$ samples 850 per skycell, a number reasonably well-matched to the density of stars 851 at most moderate Galactic latitudes. This scale is far too large to 852 track the fine-grained changes apparent in the stack images. 856 location of an object needs to determine the PSF variations present in 857 the \ippstage{stack} image, or the measurement will be somewhat 858 degraded. The highly textured PSF variations make this a very 859 challenging problem: not only would such a PSF model need to be highly 860 fine-grained, there would likely not be enough PSF stars in a given 861 \ippstage{stack} image to determine the model at the resolution 862 required. The IPP photometry analysis code uses a PSF model with 2D 863 variations using a grid of at most $6\times 6$ samples per skycell, a 864 number reasonably well-matched to the density of stars at most 865 moderate Galactic latitudes for the PS1 3$\pi$ depths. This scale is 866 far too large to track the fine-grained changes apparent in the stack 867 images. 853 868 854 869 Thus PSF photometry as well as convolved galaxy models in the stack … … 865 880 individual warp images used to generate the stack. This 866 881 \ippstage{fullforce} analysis is performed on all warps for a single 867 skycell and filter as a single unit, as this matches the arrangement 868 of the input source catalog from the \ippstage{skycal} stage. When 869 processing is queued for this stage, an entry is added to the 870 \ippdbtable{fullForceRun} primary database table linking to the 871 specific \ippdbcolumn{skycal\_id} entry that will be used as the 872 catalog for the photometry. The \ippdbcolumn{warp\_id} values for the 873 input \ippstage{warp} stage images that contributed to the 874 \ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are 882 skycell and filter as a single unit within the processing database, 883 while individual warps are processed individually in parallel as 884 separate processing jobs. 885 886 When processing is queued for this stage, an entry is added to the 887 \ippdbtable{fullForceRun} primary database table with a reference to 888 the corresponding stack and \ippdbcolumn{skycal\_id} entry that is the 889 input source of detections to be measured. The \ippdbcolumn{warp\_id} 890 values for the input \ippstage{warp} stage images that contributed to 891 the \ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are 875 892 then added to the \ippdbtable{fullForceInput} table, linked to the 876 893 primary table by the \ippdbcolumn{ff\_id} identifier. The individual … … 884 901 the same stars for all warps to the extent possible (PSF stars which 885 902 are excessively masked on a particular image are not used to model the 886 PSF). \note{this doesn't seem correct, as each warp is run 887 independently. EAM: not true!} The PSF model is fitted to all of the known source 888 positions in the warp images. Aperture magnitudes, Kron magnitudes, 889 and moments are also measured at this stage for each warp. Note that 890 the flux measurement for a faint, but significant, source from the 891 stack image may be at a low significance (less than the $5\sigma$ 892 criterion used when the photometry is not run in this forced mode) in 893 any individual warp image; the flux may even be negative for specific 894 warps. When combined together, these low-significance measurements 895 will result in a signficant measurement as the signal-to-noise 896 increases by the square root of the number of measurements. 903 PSF). The PSF model is fitted to all of the known source positions in 904 the warp images. Aperture magnitudes, Kron magnitudes, and moments 905 are also measured at this stage for each warp. Note that the flux 906 measurement for a faint, but significant, source from the stack image 907 may be at a low significance (less than the $5\sigma$ criterion used 908 when the photometry is not run in this forced mode) in any individual 909 warp image; the flux may even be negative for specific warps. When 910 combined together, these low-significance measurements will result in 911 a signficant measurement as the signal-to-noise increases by the 912 square root of the number of measurements. \note{The individual warp 913 measurements are combined together to generate averages values within 914 DVO.} 897 915 898 916 Upon completion of the forced photometry (for point sources as well as … … 901 919 that combination of \ippdbcolumn{ff\_id} and \ippdbcolumn{warp\_id}. 902 920 Once all of the entries in the \ippdbtable{fullForceInput} table have 903 finished, a summary operation is run to generate an appropriate 904 average value for each measurement, by combining the measurements from 905 each of the inputs. The output catalogs listed in the 906 \ippdbtable{fullForceResult} table are passed to the 921 finished, a summary operation is run to combine the galaxy photometry 922 analysis measurements into a single value. The output catalogs listed 923 in the \ippdbtable{fullForceResult} table are passed to the 907 924 \ippprog{psphotFullForceSummary} to do this averaging. \note{describe 908 925 what is done} When this completes, an entry is added to the … … 1031 1048 entry as such. 1032 1049 1033 \section{Post-Processing : Database Ingest \&Calibration}1050 \section{Post-Processing : Database Ingest and Calibration} 1034 1051 \label{sec:postprocessing} 1052 1053 \begin{verbatim} 1054 DVO section outline or list of topics: 1055 1056 * schema overview [ignoring sky partitioning] 1057 * measurements -> objects 1058 * images 1059 * object definition 1060 * tables in detail 1061 * adding other data types (2mass, etc) 1062 * storage details 1063 * FITS 1064 * compressed FITS 1065 * sky partitioning 1066 * parallelized DVO 1067 * addstar / ingest process [stage -> this goes elsewhere] 1068 * dvo shell description? 1069 \end{verbatim} 1035 1070 1036 1071 \subsection{DVO} … … 1398 1433 \note{Default to pointing to Flewelling et al 2017?} 1399 1434 1400 \subsection{PSPS Load \&Merge}1435 \subsection{PSPS Load and Merge} 1401 1436 \label{sec:psps} 1402 1437 \note{Default as well to pointing to Flewelling et al 2017?} 1403 1438 1404 \section{Operations \&Automation}1405 1406 \subsection{Pantasks \&Parallel Processing}1439 \section{Operations and Automation} 1440 1441 \subsection{Pantasks and Parallel Processing} 1407 1442 \label{sec:pantasks} 1408 1443 … … 1414 1449 the logical links to relate the results of one analysis stage to 1415 1450 another. In order to make a complete system which can run 1416 automatically, it is necessary to have a processwhich can use the1451 automatically, it is necessary to have a software system which can use the 1417 1452 contents of the processing database to generate the commands 1418 1453 corresponding to the analysis stages. This system needs to (1) 1419 1454 regularly examine the database to find items from stages which are 1420 ready to be processed, (2) tohave rules which define how to construct1421 the appropriate commands, (3) tocause those commands to be executed1422 within the processing system, (4) tomonitor the active processing1423 jobs for completion, and (5) tocheck on the results of those1455 ready to be processed, (2) have rules which define how to construct 1456 the appropriate commands, (3) cause those commands to be executed 1457 within the processing system, (4) monitor the active processing 1458 jobs for completion, and (5) check on the results of those 1424 1459 commands and update the processing database as needed. Within the 1425 1460 Pan-STARRS IPP, the top-level management of these operations is … … 1430 1465 might be run and to regularly generate new commands based on that 1431 1466 concept. The ``tasks'' are defined using the opihi scripting language 1432 (also shared by DVO and other user-interative programs 1433 IPP). 1467 (also shared by DVO and other user-interative programs within the 1468 IPP). 1434 1469 1435 1470 Pantasks repeatedly checks each task in an attempt to generate a new 1436 command: we say pantasks attempts to 'execute' the task. Tasks may 1437 specify the time between execution attempts, with a 1 second default. 1471 command: we say pantasks attempts to 'execute' the task in each of 1472 these attempts. Tasks may specify the time between execution 1473 attempts, with a 1 second default. 1438 1474 1439 1475 Each task must at a minimum define a command to generate. Commands … … 1441 1477 command is explicity defined in the task block (see code example in 1442 1478 Figure~\ref{fig:task_example1}) and is identical each time the task is 1443 execute . For a task with a dynamic command, the command is defined1444 within a special block of the task, called \code{task.exec}. This 1445 block is a snipet of code (in the opihi language) which is run when 1446 the task is executed. The \code{task.exec} code may refer to 1447 variables or other data structures defined by the opihi langage within1448 the pantasks environment. Within a single \ippprog{pantasks} 1449 instance, all opihi variables and data structures have global context 1450 (\ie, all are visible to all tasks). Variables are by default global, 1451 but within the context of an opihi macro (equivalent of a function 1452 call), variables may be locally-scoped. Other data structures (see 1453 be low) are global and must be protected with name space choices.1454 1455 Within the \ippprog{task.exec} macro, at some point the command tobe1456 run must be defined with the function 'command'. Once the 1457 \ippprog{task.exec} macro exits successfully, the command is the added 1458 to the list of jobs to be run within the UNIX environment. Jobs1459 may be run in one of two ways: locally or via the parallel processing 1460 system. The task, or the \ippprog{task.exec} macro, uses the 'host' 1461 command to define how to run the job. If the host is set to 'local', 1462 the n the job is run in the background by pantasks itself (using the1463 C \code{execvp} function). Otherwise, the job is sent to the parallel 1464 processing system to be run on another machine within the cluster. 1465 If the host is set to the special value 'anyhost', then the parallel 1466 processing system is allowed to choose the processing computer 1467 arbitrarily. Any other value is taken to be the DNS name of the 1468 computer on which this job should run. If the option \code{-required} 1469 is supplied to the \code{host} command, then the parallel processing 1470 system must ensure that the job only runs on the specifically named 1471 system. Otherwise, the parallel processing system may chooseto1472 redirect the command to another computer (based on whatever rules are 1473 defined for the parallelprocessing system).1474 1475 When the \ippprog{task.exec} macro is run, the code may choose (e.g., based1476 on tests of some global variables) to exit the macro with an error 1477 condition, e.g., with the 'break' command. In this circumstance, no 1478 job is produced by the task. The task will try again the next time it 1479 is executed. This feature allows for the user to set processing 1480 blocks which depend on some external tests. For example, some task 1481 may check external network connectivity and set a1479 executed. A dynamic command is defined within a special block of the 1480 task, called \code{task.exec}. This block is a snipet of code (in the 1481 opihi language) which is run each time the task is executed. The 1482 \code{task.exec} code may refer to variables or other data structures 1483 defined by the opihi language within the pantasks environment. Within 1484 a single \ippprog{pantasks} instance, all opihi variables and data 1485 structures have global context (\ie, all are visible to all tasks). 1486 Variables are by default global, but within the context of an opihi 1487 macro (equivalent of a function call), variables may be 1488 locally-scoped. Other data structures (see below) are global and must 1489 be protected with name space choices. 1490 1491 Within the \ippprog{task.exec} macro, the command to be run must be 1492 defined with the function 'command'. Once the \ippprog{task.exec} 1493 macro exits successfully, the defined command is the added to the list of jobs 1494 to be run within the UNIX environment. Jobs may be run in one of two 1495 ways: locally or via the parallel processing system. The task, or the 1496 \ippprog{task.exec} macro, uses the 'host' command to define how to 1497 run the job. If the host is set to 'local', then the job is run in 1498 the background by pantasks itself (using the C \code{execvp} 1499 function). Otherwise, the job is sent to the parallel processing 1500 system to be run on another machine within the cluster. If the host 1501 is set to the special value 'anyhost', then the parallel processing 1502 system is allowed to choose the processing computer arbitrarily. Any 1503 other value is taken to be the DNS name of the computer on which this 1504 job should run. If the option \code{-required} is supplied to the 1505 \code{host} command, then the parallel processing system must ensure 1506 that the job only runs on the specifically named computer. Otherwise, 1507 the parallel processing system may choose to redirect the command to 1508 another computer (based on whatever rules are defined for the parallel 1509 processing system). 1510 1511 When the \ippprog{task.exec} macro is run, the code may choose (e.g., 1512 based on tests of some global variables) to exit the macro with an 1513 error condition, e.g., with the 'break' command. In this 1514 circumstance, no job is produced by the task. The task will be tried 1515 again the next time it is executed. This feature allows for the user 1516 to set processing blocks which depend on some external tests. For 1517 example, some task may check external network connectivity and set a 1482 1518 variable based on the network status; other tasks may then choose to 1483 1519 wait until the network is available before attempting to run. … … 1486 1522 behavior in detail. Note that the options below may be dynamically 1487 1523 reset by the \ippprog{task.exec} macro. 1524 1525 \note{this section probably has too much detail; move this into an 1526 online user guide?} 1488 1527 1489 1528 The option 'npending' may be used to limit the number of jobs which … … 1532 1571 across many machines in the computing cluster. The parallel 1533 1572 processing system used by \ippprog{pantasks} is an independent 1534 software system (and alternatives are possible). The default parallel 1535 processing system is a program called \ippprog{pcontrol}. 1573 software system. The default parallel processing system is a program 1574 called \ippprog{pcontrol}\footnote{Alternatives are possible: e.g., 1575 {\tt condor} has been experimentally integrated with 1576 \ippprog{pantasks} for tests}. 1536 1577 1537 1578 This program is based on the same \ippprog{opihi} shell language used 1538 1579 by \ippprog{pantasks}. The two programs communicate via a shared set 1539 1580 of pipes: \ippprog{pantasks} sends commands to the standard input of 1540 the \ipppro c{pcontrol}, and accepts back responses on the standard1581 the \ippprog{pcontrol}, and accepts back responses on the standard 1541 1582 output and standard error. 1542 1583 1543 \ippprog{pcontrol} maintains a list of jobs and a list of hosts, 1544 computers on which a job could be run. Jobs may have one of several 1545 states: pending (ready to run), running, jobs which are 1546 running 1584 \ippprog{pcontrol} maintains a list of jobs (commands to be run) and a 1585 list of hosts (computers on which a job could be run). Jobs may have 1586 one of several states: pending (ready to run), running (jobs which are 1587 running), exit (job has completed), busy (job is being checked by 1588 \ippprog{pcontrol}), crash (job has exited with a signal(?), normally 1589 segv). 1590 1591 Similarly, the hosts may also have one of several states: off, down, 1592 busy, idle, etc. A single host can accept a single job at a time. 1593 Multiple hosts instances corresponding to the same machine may be 1594 specified allowing a single computer to run more than one simultaneous 1595 job. 1596 1597 During operation, pcontrol accepts new jobs from pantasks and adds 1598 them to the list of jobs to execute. It also accepts from pantasks 1599 the names of computers on which it is allowed to run those jobs. 1600 1601 When pcontrol is provided with the name of a computer, it will attempt 1602 to make an connection to that machine via ssh (or rsh?). When a 1603 connection is made, the remote shell is used to run a special 1604 interface program call \ippprog{pclient}. This program accepts 1605 command lines from pcontrol and is responsible for executing the 1606 individual commands in the local shell environment. A single ssh 1607 connection to a remote host keeps a single pclient shell running for a 1608 somewhat arbirarly long time, excuting many shell commands as needed. 1609 This architecture avoids wasting overhead making the ssh connection to 1610 the remote machine each time a command is executed, allowing for rapid 1611 excution of many commands. As a result, a single job within the IPP 1612 architecture is allowed to be very light and short running if needed. 1613 1614 After pcontrol sends a job (commands) to a specific pclient, it checks 1615 back occasionally to see if the command has been run and executed. If 1616 it has finished, then pcontrol will query for the exit status, the 1617 standard output and standard error streams from the command. (where 1618 do these go, back to pantasks?), with the results associated with the 1619 job statistics. At that point, the pclient on the remote machine is 1620 ready to accept a new job from pcontrol. If any jobs are pending in 1621 the list of jobs known to pcontrol, it will send those jobs to any 1622 machines which are idle. 1623 1624 While pcontrol interacts with the many remote machines, it 1625 occasionally interacts with pantasks to report the results from the 1626 jobs it has been monitoring. Pantasks occasionally requests a list of 1627 the completed jobs. It then requests the status information for each 1628 completed job, including the standard error and standard output. As 1629 pantasks receives this completion information, the jobs are removed 1630 from the list managed by pcontrol. Thus pcontrol maintains at most a 1631 modest list of jobs which are 'in flight', leaving all interpretation 1632 work to pantasks. 1633 1634 At the pantasks level, the tasks define how pantasks should use the 1635 exit status and output products from each job. For example, the 1636 stderr and stdout may be specified to go to a file (with static name 1637 or name dependent on the specific job). The task may define different 1638 behavior depending on the exit code from the job. 1547 1639 1548 1640 \note{discuss pclient} … … 1554 1646 1555 1647 \end{verbatim} 1556 1557 These commands may the be run by the pantasks program1558 directly, or they may be passed to another program which1559 1560 * task vs job1561 1648 1562 1649 \begin{figure} … … 1572 1659 end 1573 1660 \end{verbatim} 1574 1575 \caption{\label{fig:peaks} Illustration of peak finding and culling peaks within a 1576 footprint. Insignificant peaks within the footprint of a brighter 1577 peak are ignored in further processing. } 1661 \caption{\label{fig:simple.static.task} Example of a simple static 1662 task in the opihi-based scripting language used by pantasks. } 1578 1663 \end{center} 1579 1664 \end{figure} 1580 1581 1582 1665 1583 1666 \note{this section needs to be re-written : pclient vs pcontrol vs pantasks} … … 1660 1743 \note{I'm not convinced this is the right place for it, but it felt like a natural extension of the ``advance''}. 1661 1744 1662 \note{wording..} 1663 Beyond the warp stage, there is no longer a single ``next'' stage into 1664 which data can be queued. Because of this, more robust methods are 1665 used to advance the data. For processing data that is actively being 1666 observed at the summit, this is handled by a set of ``nightly 1667 science'' tasks and an associated \ippmisc{ippScript}. The goal of 1668 these tasks is to ensure that exposures are correctly paired into sets 1669 of\ippstage{warp}-\ippstage{warp} difference images, and that nightly1745 \note{wording..} Beyond the warp stage, there is no longer a single 1746 ``next'' stage into which data can be queued. Because of this, more 1747 sophisticated methods are used to advance the data. For processing 1748 data that is actively being observed at the summit, this is handled by 1749 a set of ``nightly science'' tasks and an associated 1750 \ippmisc{ippScript}. The goal of these tasks is to ensure that 1751 exposures are correctly paired into sets of 1752 \ippstage{warp}-\ippstage{warp} difference images, and that nightly 1670 1753 stacks are generated for MD fields. 1671 1754 … … 1742 1825 \subsection{Nebulous} 1743 1826 \label{sec:nebulous} 1744 Storing the large volume of data that is generated by the GPC1 camera 1745 was recognized early in the Pan-STARRS project as a major concern. 1746 The \ippprog{Nebulous} system was designed to organize this data. The 1747 main components of this system are a database storing the locations of 1748 the files, with a Simple Object Access Protocol (SOAP) interface 1749 between the database and the other IPP programs \note{define / mention 1750 http}. The actual files are stored on a collection of computers 1751 with substantial disk partitions in the IPP cluster, shared within the 1752 cluster via NFS. This distribution of files is useful to balance the 1753 disk I/O, as this parallelizes the load. 1827 1828 A major concern recognized early in the Pan-STARRS project is the 1829 challenge of storing and managing the large volume of data that is 1830 generated by the GPC1 camera. The \ippprog{Nebulous} system was 1831 designed to organize this data. The main components of this system 1832 are a database storing the locations of the files, with a Simple 1833 Object Access Protocol (SOAP) interface between the database and the 1834 other IPP programs \note{define / mention http}. The actual files are 1835 stored on a collection of computers with substantial disk partitions 1836 in the IPP cluster, shared within the cluster via NFS. This 1837 distribution of files is useful to balance the disk I/O, as this 1838 parallelizes the load. 1754 1839 1755 1840 The original design of \ippprog{Nebulous} was intended to aid in the
Note:
See TracChangeset
for help on using the changeset viewer.
