IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 69 and Version 70 of Processing


Ignore:
Timestamp:
Jul 2, 2010, 8:05:57 AM (16 years ago)
Author:
rhenders
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Processing

    v69 v70  
    55This page outlines the procedures and responsibilities for the person currently acting as 'IPP Processing Czar'. In a nutshell, these include:
    66
    7  * monitoring the various pantasks servers running on the production cluster using {{{pantasks_client}}}
     7 * monitoring the various pantasks servers running on the production cluster using [http://ipp004.ifa.hawaii.edu/ippMonitor/czartool_labels.php Czartool] and {{{pantasks_client}}}
     8 * keeping a close eye on the 'stdscience' pantasks server in particulr, using [http://ipp004.ifa.hawaii.edu/ippMonitor/czartool_labels.php Czartool]
    89 * alerting the IPP group to any notable errors or failures
    910 * keeping an eye on production cluster load using [http://ganglia.pan-starrs.ifa.hawaii.edu/?r=hour&s=descending&c=IPP%2520Production Ganglia]
     
    1112 * keeping an eye on available disk space using the {{{neb-ls}}} command (on any production machine)
    1213
    13 = Setup and available resources =
    14 
    15 You will need to have ipp user access on the production cluster. For convenience, have someone who already has access (anyone on the IPP team) to add your ssh public key to {{{~ipp/.ssh/authorized_keys}}}. Mostly, you will be logged into a production cluster machine as {{{ipp}}} using the {{{pantasks_client}}} program to monitor operations, however there are other useful resources.
    16 
    17  * [http://ipp004.ifa.hawaii.edu/ippMonitor/czartool_labels.php Czartool]
    18  * [http://ganglia.pan-starrs.ifa.hawaii.edu/?r=hour&s=descending&c=IPP%2520Production Ganglia] - for monitoring load on production cluster machines
    19  * [http://ipp004.ifa.hawaii.edu/ippMonitor/Login.php ippmonitor] - a window onto the gpc1 database, particularly the [http://ipp004.ifa.hawaii.edu/ippMonitor/nightSummary.php NightSummary] page
    20  * [wiki:PS1_DataRequestManagement processing priorities] - the current list of priorities, for use when setting up labels in {{{stdscience}}}
     14'''NB''' You will need to have {{{ipp}}} user access on the production cluster. For convenience, have someone who already has access (anyone on the IPP team) to add your ssh public key to {{{~ipp/.ssh/authorized_keys}}}.
    2115
    2216= Getting started and checking processing status =
    2317
    24 Log in as {{{ipp}}} user on any production cluster machine and run
    25 
    26 {{{
    27 ./check_system.sh
    28 }}}
    29 
    30 This lists the various {{{pantasks}}} servers currently running on the cluster, eg
    31 
    32 {{{
    33 pantasks server addstar is running (host: ipp004)
    34 pantasks server cleanup is running (host: ippc07)
    35 pantasks server detrend is NOT running (host: ippc06)
    36 pantasks server distribution is running (host: ippc15)
    37 pantasks server pstamp is running (host: ippdb02)
    38 pantasks server publishing is running (host: ippc08)
    39 pantasks server registration is running (host: ippc02)
    40 pantasks server replication is running (host: ippdb00)
    41 pantasks server stdscience is running (host: ippc16)
    42 pantasks server summitcopy is running (host: ippc01)
    43 }}}
     18[http://ipp004.ifa.hawaii.edu/ippMonitor/czartool_labels.php Czartool] makes it relatively easy to check the overall status of the processing pipeline. You can check the status of the various {{{pantasks_servers}}}, how much data was taken at the summit and has been copied to the cluster, and the status of various processes within {{{stdscience}}}, chip, camera, warp, diff etc.
    4419
    4520= Using {{{pantasks}}} =
    4621
    47 There are numerous {{{pantasks}}} servers, as shown above. To use {{pantasks_client}}, first move to the directory corresponding to the server of interest, eg {{{~ipp/stdscience/}}}, then run
     22There are numerous {{{pantasks}}} servers. Their status can be checked with [http://ipp004.ifa.hawaii.edu/ippMonitor/czartool_labels.php Czartool], but it is often necessary to use a client directly. To do this {{pantasks_client}}, first move to the directory corresponding to the server of interest, which are all under {{{~ipp}}} on any cluster machine, for example {{{~ipp/stdscience/}}}, then run
    4823
    4924{{{
     
    9671
    9772
     73
    9874{{{
    9975ns.show.dates