| | 31 | = Pantasks configuration = |
| | 32 | |
| | 33 | There are currently two pantasks_servers defined for the ipplanl user. The first is the stdlocal (host j001b04) pantasks, which has been working on the PV3 pole. This is doing entirely local processing, and almost certainly needs more processing power devoted to it (I haven't had time to attempt to balance this and the nightly processing). I had originally intended this to be phased out once the pole is finished, but it may make more sense to transfer the local processing tasks from the second pantasks as well. |
| | 34 | |
| | 35 | The second pantasks is stdlanl (host ippc11), and has been running the remote tests. In addition to the remote jobs, it also is running stage-advance tasks, as well as the fake and warp overlap jobs. These are needed to synchronize the database and ensure that jobs that are ready to be bundled have all the necessary metadata. As this pantasks is the one that submits jobs for processing, it must be running on a host that has a MasterControl ssh connection established to wtrw. Otherwise, remote jobs that attempt to run will be pushed to a state of "auth". |
| | 36 | |
| | 37 | = SSH connections = |
| | 38 | |
| | 39 | I've written two perl scripts that attempt to make managing the SSH connection easier. {{{lanl_control_tool.pl [start|stop|check]}}} can be used to start, stop, and check the status of the ssh connection, by issuing the appropriate command. {{{lanl_switch_users.pl [username]}}} changes the .ssh/config symlink to point to the config file for a different username. As discussed on the [http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/External_Supercomputing_Notes#a2014-07-02 remote development wikipage], reconnecting will follow the steps: |
| | 40 | |
| | 41 | 1. Log in as ipplanl user on main node (currently ippc11). |
| | 42 | 2. Check status of ssh control master connection: {{{./lanl_control_tool.pl}}} If the connection is down, skip to #5. |
| | 43 | 3. Stop network active pantasks tasks: (this needs to have a control script) {{{remote.poll.off; remote.exec.off}}}. Wait for outstanding *.run jobs to terminate. This may take a long time. |
| | 44 | 4. Terminate the running master connection: {{{./lanl_control_tool.pl stop}}}. |
| | 45 | 5. Confirm that the correct .ssh/config file will be read: {{{./lanl_switch_users.pl <lanl_username>}}} |
| | 46 | 6. Reinitialize master connection: {{{./lanl_control_tool.pl start}}}. Enter password at prompt. This will start the connection, and then push it into the background, returning control of the terminal. |
| | 47 | 7. Restart network active pantasks tasks: {{{remote.poll.on; remote.exec.on}}} |
| | 48 | 8. Revert any paused jobs: {{{remotetool -dbname gpc1 -revertauth -label <PV3_processing_label>}}} |
| | 49 | |
| | 50 | This reconnection is necessary as the lanl end will disconnect the ssh connection after 12 hours. By setting up the MasterControl connection periodically, we can run jobs remotely through that main connection without having to authenticate each login session. |
| | 51 | |
| | 52 | = System overview = |
| | 53 | |
| | 54 | Given a label, the remote.define task periodically attempts to find state=new exposures for a particular stage, and construct a remoteRun that links those exposures into a single processing bundle. There is logic to prevent the same stage/stage_id pair from going into two different remoteRuns, and I'm fairly confident that that's working correctly. The warp stage has a separate define as the extra processing (fake and overlap) slows down how quickly a warp is active, and so warp has a longer poll time. The remoteRun has a state of 'new' at this point. |
| | 55 | |
| | 56 | Once a run is defined, the stage appropriate prepare script is executed to construct the command list (OUTROOT.config), lists of symbolic and disk file names on both local and remote ends (OUTROOT.{generate,return,transfer,check}, and the actual msub command (OUTROOT.cmd) containing the resource request that will be executed. These prepare scripts take a long time to finish, but as we can run lots of them in parallel, this shouldn't be a problem. Once the prepare script finishes, the remoteRun moves to a state of 'pending'. |
| | 57 | |
| | 58 | The exec task sends the prepared output to lanl, and verifies that the required files exist on the remote end. If not, these are transferred from the local copy. Once that completes, the job bundle is submitted into the remote scheduler. The scheduler returns a job_id which is inserted into the remoteRun, and the state is set to 'run'. |
| | 59 | |
| | 60 | After an hour (based on the last_poll value in the database), the poll mode of the exec script is called for the remote_id/job_id, and the remote system is probed to see if the job has finished. If the job isn't done, the last_poll is updated and the script exits. If the ssh connection isn't active, the state is set to 'auth', and will need to be reverted back to 'run'. If the job is finished, the generated output is returned to the local side. The generated outputs include dbinfo files, which contain the appropriate chiptool/camtool/warptool/stacktool commands to add the individual component database information. These are used to gauge what jobs succeeded, as a failure (due to crash, remote scheduler killing the job, etc) will prevent the dbinfo file from being constructed. If a dbinfo file is missing after the data is transferred back, then that stage_id is removed from the remoteComponent table, allowing the exposure to be reattempted in a future job. |
| | 61 | |
| | 62 | = How things go wrong = |
| | 63 | |
| | 64 | In the development process, it seems like things go wrong in the following ways. |
| | 65 | |
| | 66 | == Chip stage == |
| | 67 | |
| | 68 | Generally this has gone smoothly. The main problem is that the prepare script takes a very long time due to the number of database interactions and nebulous dereferencing. This also has a long exec time due to the need to transfer raw OTA/detrend files. |
| | 69 | |
| | 70 | == Camera stage == |
| | 71 | |
| | 72 | Also seems to work smoothly. This transfers more data back than chip does (due to the SMF sizes), but hasn't hit the HPN-SSH bug in the tests. There is evidence of "fracturing," where a large chip remoteRun will take many camera stage remote.define cycles, resulting in a single chip remoteRun ending up in multiple smaller camera remoteRuns. This is largely a logistical issue, which I hope will be improved as the number of concurrent jobs increases. |
| | 73 | |
| | 74 | == Warp stage == |
| | 75 | |
| | 76 | Processing had some issues due to being cut off my the wall-time limit in the remote scheduler. This is a result of the large numbers of files constructed at this stage that we wish to transfer home. Due to differences in symbolic/disk filename, each file generated needs to be symlinked into a dummy disk hierarchy on the remote side to aid in the transfer. The overhead of constructing these symbolic links is about 40% of the total job execution time, resulting in a dramatic underestimate of the required processing time. This estimate has been revised, and seems match reality better. |
| | 77 | |
| | 78 | However, the large number of returned files has resulted in us hitting the HPN-SSH bug. Briefly, we're using HPN patches on ssh, which allow us to send unencrypted data in filetransfers after validating our credentials. This results in a significant speed increase. However, the ssh connection overallocates a buffer, resulting in the connection from lanl to ippc2X to be dropped. I have tried all the options that the documentation discusses, but this does not reliably solve this issue. This results in the transfers failing, and this in turn led to the exposures being removed and requeued. The current exec command should not remove and requeue exposures until it detects that the entire job failed (not the best solution, but better than wasting cycles re-warping things). |
| | 79 | |
| | 80 | == Stack stage == |
| | 81 | |
| | 82 | Previous stack jobs have worked correctly, although this hasn't been cleanly tested for the ipp-pv3-20140717 tag. These jobs are computationally smaller as each stack_id corresponds to a single skycell and not a full exposure. In addition, the threading and number of jobs per host need to be decreased compared to prior stages, as the memory footprint of a ppStack is larger than anything else. The remote compute nodes do not have swap space, so if the memory is exhausted, it results in a system crash and the scheduler terminates the job. |
| | 83 | |
| | 84 | = PV3 plan = |
| | 85 | |
| | 86 | The pole will be completed locally at the IPP, as the exposure overlap is large and would result in files being transferred back to lanl for processing. Once this is finished, the rest of the sky can be lauched with the standard LAP apparatus. This will queue chips and stacks, which should feed directly into the remote processing define/prepare/exec/poll system. |
| | 87 | |
| | 88 | The LAP queue files have been pre-defined for the entire sky, and are located in {{{/data/ippc18.0/home/ipplanl/queues/lap}}}. |
