Details

I've censored certain details in the description below both to prevent issues with security and to make things somewhat easier to understand. A description of each detail parameter is listed in the table below. For single permanent values, I've used all caps, and for example parameters, I've used the @TEMP_FILE@ convention.

DMZ_HOST	Public internet connected computer that shields the computing nodes.
SEC_HOST	Secure internal front end to the computing nodes. Located beyond DMZ_HOST.
DMZ_DN	Full domain name for the DMZ_HOST
USERNAME	Cluster username
IPP_PATH	Path to IPP build

Logging in

To reduce the number of login/password authorizations, it's recommended to use the ControlMaster ssh parameters. This requires only the first connection to be validated, as subsequent connections are passed through the already validated connection. The .ssh/config options to implement this are:

Host DMZ_HOST
  User          USERNAME
  Hostname      DMZ_DN
  ForwardX11    yes
  ForardAgent   yes
  RequestTTY    force
  ControlMaster auto
  ControlPath   ~/.ssh/connections/%h_%p_%r

The ~/.ssh/connections/ directory needs to be created if it does not already exist. With this configuration in place, you can log in to the computing front end with ssh -t DMZ_HOST ssh SEC_HOST. The DMZ_HOST does not support commands other than ssh and scp, and is used only to bounce the connection to SEC_HOST.

Data transfer

As the DMZ_HOST does not have available disk space, data transfers need to write directly to the SEC_HOST disks. The following commands illustrate how to use scp to transfer files to and from SEC_HOST:

scp @LOCAL_FILE@ DMZ_HOST:SEC_HOST:/@PATH_TO_DESTINATION@/
scp DMZ_HOST:SEC_HOST:/@PATH_TO_FILE@/@REMOTE_FILE@ @LOCAL_DESTINATION@

Data transfer connections need to be established from outside, as SEC_HOST cannot see the public internet.

SEC_HOST storage locations

There are four main storage locations.

homedir	Small, and should not be used for anything that can be placed elsewhere.
/usr/projects/ps1/	This should be the default storage location for non-transient data products. It should have in the future more disk space than it currently reports (20GB).
/scratch/USERNAME/	A very large single volume that is automatically cleaned on timescales of weeks.
/scratch3/USERNAME/	Same.

IPP build

Due to the data transfer limitations, svn co will not work on SEC_HOST. A copy of the code needs to be pushed into place. However, a slightly old (r35169 2013-02-14) version of IPP is available, which I was able to get to work simply by adding

alias psconfig "source IPP_PATH/psconfig.csh"
psconfig default

in my .cshrc.

IPP Running

There is no database running on SEC_HOST, which impacts how the IPP can be run. The ippScripts assume a database exists that can be probed to determine what should be executed. In addition, the detrend information is stored in the database, which means processing raw images will need to have detrends specified manually.

Despite this issue, I was able to test that the IPP build does work. Running ppImage

> ppImage -file o5303g0240o.ota67.fits test.out -recipe PPIMAGE CHIP -Db PHOTOM F -Db DARK F -Db FLAT F -Db NOISEMAP F -threads 1 -trace config 89 -Db MASK F -Db NONLIN F -log test.log -tracedest test.trace
Number of leaks to display: 500

produced all expected outputs from the recipe (taking into account the command line option changes). For this test, I disabled the detrending manually to avoid having to specify a series of -mask/-dark/-flat options.

Parallelization

The supercomputing resources are managed by Moab, for which I've been using ​this page as documentation. Briefly, a job script is constructed that outlines the resources required for completion, and this is submitted to the moab scheduler. The resources are allocated, the job is run, and the job terminates. This is significantly different than the standard IPP scheduling system, which assumes full access to the computing hardware.

stask

The example stask scripts seem to provide a path around this limitation. From my reading of the scripts:

1. A call to the stask.py python script is defined in the stask shell script. This script also defines a task list. stask is then submitted to moab as a job.
2. The moab job constructs a list of nodes allocated to the job.
3. The moab job passes this node list and the task list to the python script.
4. The python script (running under moab) connects to the individual node via ssh, changes to the appropriate workdir, and runs the shell script mult.sh with the task parameters.
5. The shell script (under python as a moab job, on the individual node) runs the requested job under the ​GNU parallel framework.
6. Python completes when the shell scripts are finished on all nodes, clearing the moab job to complete and release the resources requested.

As of 2013-12-18, I have not run an IPP command through this stask scripts.

comments by EAM

Issues which we need to consider for large-scale processing on this cluster:

• lack of a database
  • how do we provide the right detrends?
  • how do we synchronize results with IfA / gpc1?
  • we cannot use the database for job sequencing (gpc1)
• interacting with Moab
  • can we use stask.py to run under pantasks or equivalent?
  • can we use pantasks to talk directly to Moab? (note that Serge wrote a pantasks backend to talk to condor as a drop-in replacement for pcontrol; the same could potentially be done for Moab)

The primary problem is that, without a database, we cannot coordinate our operations as we currently do. Even if pantasks could talk to Moab, without a database, we cannot sequence and schedule jobs. This precludes the use of a straight build of IPP locally to run the full system.

In any arrangement, we will certainly want to ship all of our detrend data and the reference photometry / astrometry database up front. We will have to define a way to automatically generate the commands and to retrieve results for some chunk of the processing, including the info that needs to be pushed into the gpc1 database.

Some possible ways of handling the interaction:

1. treat the remote cluster as a set of nodes and use our pantasks (talking directly to gpc1) to send the jobs. This would be a very fine-grained integration with the current IPP processing. It would require any job (say chip) to generate a command and a bundle with the data and the pointers to the appropriate local references (detrends, etc). then the bundle would be shipped to the remote cluster. when the job is done, that fact needs to be carried back to the local pantasks, and the results retrieved.
2. identify a complete sequence (say, chip, cam, warp) and send that as a bundle. This sounds easier that #1 above but is in fact equivalent in terms of the need for automatic generation of the bundle / command, shipping the data and discovering the completion
3. choose a large sequence (eg, all chip processing and all downstream processing to stack a complete projection cell). this could be done with less automation at least at first, though there are still ~2k projection cells per filter, so some automation would be needed eventually.

It seems to me that any solution is going to require us to automatically ship data on some scale to LANL, automatically send a job to Moab, and automatically capture the result. We should start on achieving those goal in a generic way rather than worrying about running IPP jobs specifically up front.

2013-12-19 Telecon

• Storage.
  • Other local storage exists there, although it is significantly slower than transferring even from Hawaii.
  • The temporary scratch disk space is cleared automatically, although there is some leeway. Files that are accessed (or updated?) have their retention time extended. There are limits to this extension, so manually gaming the system is undesirable.
• Network.
  • SVN access may be possible with exceptions in the firewall. These exceptions will be required to point to only a small number of IP addresses.
  • Data transfer can be done through an intermediate node, although this is limited to only 20TB of temp space. This suggests pushing (as listed above) or having a firewall hole to pull data through is a more workable solution.
  • ​HPN-SSH exists, and can be used to speed up scp transfers (I'm not fully clear on the details, but the claim is up to 10x speed improvement).
• We do have an account for moab processing, although by default our jobs should be "charged" to the correct one. The allocation is sufficiently large that this isn't much of a concern.
• Bundling tasks.
  • Due to the lack of database at the computing center, it's likely that we'll need to bundle a command list here, transfer that list along with the associated data, and then retrieve results.
  • Doing this with the SAS is probably the most appropriate first test.

Bundling levels

• C-level. This would require script options that instead of executing the final command (ppImage/psastro/pswarp/etc), those commands would be written to an output file and the script would terminate. As we do database updates after that command, we would need to also write out a set of completion commands to update the database with the result.
  • This requires significant changes in the current scripts.
  • It may be difficult to keep the database synchronized at all. Once a task has been sent to the supercomputing cluster, we need a way to flag the job as "executing" so we don't launch a second run. The alternative is that the scripts will simply wait until the job is done.
  • Keeping all the database interactions in Hawaii, and just letting the computational part execute in the supercomputer gives a solution to the detrend issue. We can construct the command to manually specify which detrend to use. Gene had the idea to fold this determination into chiptool, such that the returned metadata includes the detrends that should be used.
• Script level. This would send entire perl script level commands to the computing cluster (chip_imfile.pl/etc), with the database interaction disabled (--no-update).
  • Although this shouldn't require major changes to the scripts, we still need something to check the results and issue the proper database commands in Hawaii.
  • This also prevents the detrend fix from above working. In fact, it would require some sort of script change to get the construction of the final command done correctly. Currently that's handled by passing only the stage_id to the script, which then asks the database for information about the object described by the stage_id.
• Stage level. An entire stage worth of final commands is constructed for a given label.
  • This solves the synchronization issue of the C-level by knowing that we'll only queue the commands once.
  • Allows for a scan of the results before generating the commands for the next stage.
  • Requires a large amount of data to be synchronized at each step, although this could be rolled over once we understand the data transfer vs. computation times.
• Project level. All stages are queued at the same time, with some logic used in moab to determine which jobs are dependent on others.
  • Even larger synchronization issues.
  • Requires some
  • Many stages take metadata objects as inputs. If some components fail permanently (i.e., single bad chip failures, poorly populated skycells), we need to not include them in the input metadata. This prevents all metadata from being pre-calculated.

2014-01-15

I've started playing around with running jobs with the MOAB scheduler, and after being completely confused by the msub/srun split, I think I've sorted out more details on how to submit jobs.

My first test was to submit a ppImage job to the Moab scheduler. Using the command msub -v ppI_test.cmd where that command script contains:

#!/bin/tcsh
##### Moab controll lines
#MSUB -l nodes=1:ppn=4,walltime=1:00:00
#MSUB -j oe
#MSUB -V
#MSUB -o /scratch3/watersc1/job.out

##### Commands
date
ppImage -file /scratch3/watersc1/o5303g0240o.ota67.fits /scratch3/watersc1/ppI_test -recipe PPIMAGE CHIP -Db PHOTOM F -Db DARK F -Db FLAT F -Db NOISEMAP F -Db MASK F -Db NONLIN F -threads 4
date

This submitted the job and executed it, saving the stderr/stdout information in the job.out file specified. This runs all the commands contained sequentially, meaning that there is no parallelization internal to the task, and all allocated resources can be used by each command in the task.

The second test used srun as the main command, allowing multiple jobs to be run in parallel, sharing the allocated resources: msub -V ppI_srun_test.cmd.

#!/bin/tcsh
##### Moab controll lines
#MSUB -l nodes=2:ppn=4,walltime=0:10:00
#MSUB -j oe
#MSUB -V
#MSUB -o /scratch3/watersc1/job_srun.out

##### Commands
date
srun -n2 -l --multi-prog ppI_srun_test.config
date

The config file for srun:

# ppI double run test config file
# srun -n2 -l --multiprog ppI_srun_test.config

0     ppImage -file /scratch3/watersc1/o5303g0240o.ota67.fits /scratch3/watersc1/ppI_srun_test_0 -recipe PPIMAGE CHIP -Db PHOTOM F -Db DARK F -Db FLAT F -Db NOISEMAP F -Db MASK F -Db NONLIN F -threads 1
1     ppImage -file /scratch3/watersc1/o5303g0240o.ota67.fits /scratch3/watersc1/ppI_srun_test_1 -recipe PPIMAGE CHIP -Db PHOTOM F -Db DARK F -Db FLAT F -Db NOISEMAP F -Db MASK F -Db NONLIN F -threads 1

I'm still looking into tracking command exit code status, but srun provides a -K option that terminates if any command exits with non-zero exit code.

Forcing an error by having command 1 look for a non-existent input fits file produces the following line in the output file:

srun: error: @HOST@: task 1: Exited with exit code 3

2014-01-23

After completing some of the most obvious scripts that are necessary to make this work, I've come up with the following processing outline that I think satisfies all of the needed steps.

Queuing/database management

All processing needs to begin with chipRun entries defined in the standard way. chiptool -pendingimfile provides the majority of the information needed, so it's best to reuse this framework. However, after the chipRun entries are defined, they need to be registered into a "supercomputeRun" database entry. My current assumption is that a chip_id or chipRun.label field will be used in the definebyquery statement. The supercomputeRun entry will manage the communication with the remote supercompute resources. This table needs to have the following fields:

• sc_id: main table index
• stage: current stage being processed for this entry
• stage_id: reference to the stageRun table
• state: in addition to the run/full/cleaned values, this will need values pending, auth, and error (described below).
• last_poll: date string indicating last time this entry was remotely polled.
• job_id: remote processing job_id.
• fault: standard fault code
• path_base: local storage location for supercompute file generation.
• job_policy: policy state for dealing with partially incomplete runs.

Processing order

Once the supercomputeRun is inserted (with state = 'new'), the initial prep task will run, doing the following:

• Construct list of required files necessary to complete the task on the remote cluster, and their local analogs (if they exist).
• Construct the command script that defines the resources needed and will launch the srun command.
• Construct the config script needed for srun to parallelize the processing.
• Update supercomputeRun to state = 'pending'

With the preparation done, the launch task runs, with the following effects:

• Check that the connection to the remote supercomputer is available. If not, set the database entry to state = 'auth', update the last_poll entry, and quit.
• Send list of required files to the remote cluster, and run the validation script on the remote end. This script will produce a file listing any missing entries. This file is returned to the local site, and a transfer list is constructed of the files that need to be pushed to the remote site.
• Any file pushes that are required are performed, and the command and config scripts are pushed as well.
• The command script is inserted into the supercomputing queue and the job_id value is obtained.
• The supercomputeRun is set to state = 'run', and the job_id value updated.
• A job poll is performed (setting last_poll), and the script either terminates (allowing a future poll to check), or sleeps until the poll determines the job has completed. A future poll will skip the previous steps and return to this point.
• The remote job status completes, and the resulting files are pulled back from the remote site, including any permanent files (log, stats, SMF, etc), as well as a job_summary compiled at the end of the command script.
• If the remote job has completed successfully, or if the job_policy is set to "ignore", the local database is updated to reflect that the stageRun has completed successfully, and the state is updated to 'full'. If the job_policy is not set to ignore (job errors are fatal), the local database is updated with information about the jobs that have completed successfully, but the jobs that have failed are marked as so. The supercomputeRun is set to have a fault != 0, and the state = 'error'. This behavior then will require manual intervention to update the database and resolve the failed jobs.

At this point, a master control process will have to step in and launch new supercomputeRun entries for the next step. Upon adding this, the chip->supercomputeRun operation is likely an operation of this master control program operating on a label basis.

Tasks/scripts/tools

• Tasks
  • scproc.pro
    • master_control.run
    • prepare.load
    • prepare.run
    • launch.load
    • launch.run
    • launch.poll
• Scripts
  • sc_prepare_chip.pl
  • sc_prepare_cam.pl
  • sc_prepare_warp.pl
  • sc_prepare_stack.pl
  • sc_validate_files.pl
  • sc_remote_exec.pl
  • sc_validate_processing.pl
  • sc_master_control.pl
• Tools
  • sctool