Remote Processing Maintenance

ipplanl scripts

• lanl_status.pl Main script to check system status.
  • lanl_status.pl Print status information.
  • lanl_status.pl reconnect Do safe checks of the system, and restart the ssh connection.
• lanl_control_tool.pl Directly start and stop ssh connection.
  • lanl_control_tool.pl stop Stop ssh connection.
  • lanl_control_tool.pl start Start ssh connection.
• lanl_switch_users.pl Change the user who is operating the ssh connection/running commands at LANL.
  • lanl_switch_users.pl USERNAME Switch to user USERNAME
• revert_stacks.sh Revert local stack processing.
  • revert_stacks.sh
• drop_bad_data.pl Print [PV3_failed_data wiki page] entries and stacktool/warptool commands for runs with unsolvable problems.
  • drop_bad_data.pl Copy appropriate wiki table entries, and copy and execute the commands listed.
• revert_lanl.sh All purpose "kick" script. No arguments. Returns:
  • Summary of IPP queue, will print correct ln -sf command to promote the next queue file when appropriate.
  • Summary of LANL queue. Same checks for new queues.
  • stat command output of files in ~/.ssh/connection/ to check the age of current ssh connection.
  • Execute remotetool -revertauth for the LANL label.
  • Execute remotetool -revertrun for the LANL label.
  • Construct list of remotetool -revertcomponent commands that need to be copied and run to clear failed sub components.
  • Generate a ~/lanlstats/mdiag report containing the current /scratch3/ df information and the mdiag processing quota. Currently disabled as this seems to cause the master SSH connection to drop at a higher than expected rate.

General operation task list

• Check that your username is the active user, using the lanl_switch_users.pl command.
• Establish the ssh connection, either with lanl_status.pl reconnect or lanl_control_tool.pl start.
• Periodically run the revert_lanl.sh script. remotetool -revertrun has a cronjob that runs once an hour, which ensures that ssh priority errors don't build up forever, but this is the only way to automatically clear failed components.
• Either wait for the ssh connection to be terminated by the remote side, or terminate it manually with lanl_control_tool.pl. Losing jobs in the middle of a transfer is not fatal, as the rsync commands pick up from the point the connection was lost.
• Check drop_bad_data.pl and revert_stacks.sh daily, or when obvious warp and stack failures build up.

Setup/authentication

1. Step one is to see if your card works correctly. Type in your Z number, and then use the card to generate a password for this webform:

​https://weblogin.lanl.gov/

That should take you to a page that tells you you've logged in. It should link to any needed training things (I think it did in the past, but I don't see anything for me there).

In any case, if you can log in to weblogin, you should be able to log into Mustang.

2. Except for the part that I forgot about. Once you've logged on to weblogin, you should see a "network registry" option in your list of web applications. Click through to that and follow the instructions to generate your username. Once that's done, we need to have Mike request accounts on the compute system (which takes about a day). This likely also puts you on the ICN mailing list where you'll see the scheduled downtime emails.

3. Log in as the ipplanl user with the same ssh key as the standard ipp user. In the .ssh directory you'll find a config.watersc1 file. Construct one for yourself with your user name, and then modify the User entry in the config file. Copy this config file (or its contents) to your personal (not ipplanl) ssh config. The ipplanl instance is there for the lanl-czaring purpose, and the personal one for you to log in manually.

Once you've made this ssh config change, you can now call:

ssh wtrw

This connects to the front end firewall computer. It does not have a real shell, so you're limited to connecting onward to the Mustang frontend:

ssh mu-fe

4. You're now on one of the four front end nodes for Mustang, any of which works. I've constructed a directory:

/turquoise/usr/projects/ps1/watersc1/user_template

Which contains the templates you should use for your .ipprc, .cshrc, and .ssh/config files. The .ssh/config is the only one that needs editing, with my username/connection directory names changed. Ensure that the connection directory exists, and that you can successfully ssh back to the cluster (as you; You'll likely need to generate/add keys).

EAM : additional steps not explicit above:

mkdir .ssh
chmod 700 .ssh
mkdir .ssh/connection

Pantasks configuration

There are currently two pantasks_servers defined for the ipplanl user. The first is the stdlocal (host j001b04) pantasks, which has been working on the PV3 pole. This is doing entirely local processing, and almost certainly needs more processing power devoted to it (I haven't had time to attempt to balance this and the nightly processing). I had originally intended this to be phased out once the pole is finished, but it may make more sense to transfer the local processing tasks from the second pantasks as well.

The second pantasks is stdlanl (host ippc11), and has been running the remote tests. In addition to the remote jobs, it also is running stage-advance tasks, as well as the fake and warp overlap jobs. These are needed to synchronize the database and ensure that jobs that are ready to be bundled have all the necessary metadata. As this pantasks is the one that submits jobs for processing, it must be running on a host that has a MasterControl ssh connection established to wtrw. Otherwise, remote jobs that attempt to run will be pushed to a state of "auth".

SSH connections

I've written two perl scripts that attempt to make managing the SSH connection easier. lanl_control_tool.pl [start|stop|check] can be used to start, stop, and check the status of the ssh connection, by issuing the appropriate command. lanl_switch_users.pl [username] changes the .ssh/config symlink to point to the config file for a different username. As discussed on the ​remote development wikipage, reconnecting will follow the steps:

1. Log in as ipplanl user on main node (currently ippc11).
2. Check status of ssh control master connection: ./lanl_control_tool.pl If the connection is down, skip to #5.
3. Stop network active pantasks tasks: (this needs to have a control script) remote.poll.off; remote.exec.off. Wait for outstanding *.run jobs to terminate. This may take a long time.
4. Terminate the running master connection: ./lanl_control_tool.pl stop.
5. Confirm that the correct .ssh/config file will be read: ./lanl_switch_users.pl <lanl_username>
6. Reinitialize master connection: ./lanl_control_tool.pl start. Enter password at prompt. This will start the connection, and then push it into the background, returning control of the terminal.
7. Restart network active pantasks tasks: remote.poll.on; remote.exec.on
8. Revert any paused jobs: remotetool -dbname gpc1 -revertauth -label <PV3_processing_label>

This reconnection is necessary as the lanl end will disconnect the ssh connection after 12 hours. By setting up the MasterControl connection periodically, we can run jobs remotely through that main connection without having to authenticate each login session.

System overview

Given a label, the remote.define task periodically attempts to find state=new exposures for a particular stage, and construct a remoteRun that links those exposures into a single processing bundle. There is logic to prevent the same stage/stage_id pair from going into two different remoteRuns, and I'm fairly confident that that's working correctly. The warp stage has a separate define as the extra processing (fake and overlap) slows down how quickly a warp is active, and so warp has a longer poll time. The remoteRun has a state of 'new' at this point.

Once a run is defined, the stage appropriate prepare script is executed to construct the command list (OUTROOT.config), lists of symbolic and disk file names on both local and remote ends (OUTROOT.{generate,return,transfer,check}, and the actual msub command (OUTROOT.cmd) containing the resource request that will be executed. These prepare scripts take a long time to finish, but as we can run lots of them in parallel, this shouldn't be a problem. Once the prepare script finishes, the remoteRun moves to a state of 'pending'.

The exec task sends the prepared output to lanl, and verifies that the required files exist on the remote end. If not, these are transferred from the local copy. Once that completes, the job bundle is submitted into the remote scheduler. The scheduler returns a job_id which is inserted into the remoteRun, and the state is set to 'run'.

After an hour (based on the last_poll value in the database), the poll mode of the exec script is called for the remote_id/job_id, and the remote system is probed to see if the job has finished. If the job isn't done, the last_poll is updated and the script exits. If the ssh connection isn't active, the state is set to 'auth', and will need to be reverted back to 'run'. If the job is finished, the generated output is returned to the local side. The generated outputs include dbinfo files, which contain the appropriate chiptool/camtool/warptool/stacktool commands to add the individual component database information. These are used to gauge what jobs succeeded, as a failure (due to crash, remote scheduler killing the job, etc) will prevent the dbinfo file from being constructed. If a dbinfo file is missing after the data is transferred back, then that stage_id is removed from the remoteComponent table, allowing the exposure to be reattempted in a future job.

How things go wrong

In the development process, it seems like things go wrong in the following ways.

Chip stage

Generally this has gone smoothly. The main problem is that the prepare script takes a very long time due to the number of database interactions and nebulous dereferencing. This also has a long exec time due to the need to transfer raw OTA/detrend files.

Camera stage

Also seems to work smoothly. This transfers more data back than chip does (due to the SMF sizes), but hasn't hit the HPN-SSH bug in the tests. There is evidence of "fracturing," where a large chip remoteRun will take many camera stage remote.define cycles, resulting in a single chip remoteRun ending up in multiple smaller camera remoteRuns. This is largely a logistical issue, which I hope will be improved as the number of concurrent jobs increases.

Warp stage

Processing had some issues due to being cut off my the wall-time limit in the remote scheduler. This is a result of the large numbers of files constructed at this stage that we wish to transfer home. Due to differences in symbolic/disk filename, each file generated needs to be symlinked into a dummy disk hierarchy on the remote side to aid in the transfer. The overhead of constructing these symbolic links is about 40% of the total job execution time, resulting in a dramatic underestimate of the required processing time. This estimate has been revised, and seems match reality better.

However, the large number of returned files has resulted in us hitting the HPN-SSH bug. Briefly, we're using HPN patches on ssh, which allow us to send unencrypted data in filetransfers after validating our credentials. This results in a significant speed increase. However, the ssh connection overallocates a buffer, resulting in the connection from lanl to ippc2X to be dropped. I have tried all the options that the documentation discusses, but this does not reliably solve this issue. This results in the transfers failing, and this in turn led to the exposures being removed and requeued. The current exec command should not remove and requeue exposures until it detects that the entire job failed (not the best solution, but better than wasting cycles re-warping things).

Stack stage

Previous stack jobs have worked correctly, although this hasn't been cleanly tested for the ipp-pv3-20140717 tag. These jobs are computationally smaller as each stack_id corresponds to a single skycell and not a full exposure. In addition, the threading and number of jobs per host need to be decreased compared to prior stages, as the memory footprint of a ppStack is larger than anything else. The remote compute nodes do not have swap space, so if the memory is exhausted, it results in a system crash and the scheduler terminates the job.

PV3 plan

The pole will be completed locally at the IPP, as the exposure overlap is large and would result in files being transferred back to lanl for processing. Once this is finished, the rest of the sky can be lauched with the standard LAP apparatus. This will queue chips and stacks, which should feed directly into the remote processing define/prepare/exec/poll system.

The LAP queue files have been pre-defined for the entire sky, and are located in /data/ippc18.0/home/ipplanl/queues/lap.