| 28 | | * Check that the connection to the remote supercomputer is available. If not, set the database entry to state = 'auth', update the last_poll entry, and quit. |
| 29 | | * Send list of required files to the remote cluster, and run the validation script on the remote end. This script will produce a file listing any missing entries. This file is returned to the local site, and a transfer list is constructed of the files that need to be pushed to the remote site. |
| 30 | | * Any file pushes that are required are performed, and the command and config scripts are pushed as well. |
| 31 | | * The command script is inserted into the supercomputing queue and the job_id value is obtained. |
| 32 | | * The supercomputeRun is set to state = 'run', and the job_id value updated. |
| | 32 | * Check that the connection to the remote cluster is available. If not, set the database entry to state = 'auth', update the last_poll entry, and quit. |
| | 33 | * Send the prepared files from the previous step. This includes the command and config scripts, as well as a list of required files needed to complete the job. The file validation script is run on the remote end and outputs any missing entries through the ssh stream. A transfer list is constructed of the files that need to be pushed to the remote site. |
| | 34 | * Any file pushes that are required are performed. |
| | 35 | * The command script is inserted into the remote processing queue and the job_id value is obtained. |
| | 36 | * The remoteRun is set to state = 'run', and the job_id value updated. |
| 35 | | * If the remote job has completed successfully, or if the job_policy is set to "ignore", the local database is updated to reflect that the stageRun has completed successfully, and the state is updated to 'full'. If the job_policy is not set to ignore (job errors are fatal), the local database is updated with information about the jobs that have completed successfully, but the jobs that have failed are marked as so. The supercomputeRun is set to have a fault != 0, and the state = 'error'. This behavior then will require manual intervention to update the database and resolve the failed jobs. |
| 36 | | |
| 37 | | At this point, a master control process will have to step in and launch new supercomputeRun entries for the next step. Upon adding this, the chip->supercomputeRun operation is likely an operation of this master control program operating on a label basis. |
| | 39 | * If the remote job has completed successfully, or if the job_policy is set to "ignore", the local database is updated to reflect that the stageRun has completed successfully, and the state is updated to 'full'. If the job_policy is not set to ignore (job errors are fatal), the local database is updated with information about the jobs that have completed successfully, but the jobs that have failed are marked as so. The remoteRun is set to have a fault != 0, and the state = 'error'. This behavior then will require manual intervention to update the database and resolve the failed jobs. |
| | 40 | |
| | 41 | At this point, a master control process will have to step in and launch new remoteRun entries for the next step. Upon adding this, the chip->remoteRun operation is likely an operation of this master control program operating on a label basis. |
| 255 | | == 2014-01-23 == |
| 256 | | |
| 257 | | After completing some of the most obvious scripts that are necessary to make this work, I've come up with the following processing outline that I think satisfies all of the needed steps. |
| 258 | | |
| 259 | | === Queuing/database management === |
| 260 | | |
| 261 | | All processing needs to begin with chipRun entries defined in the standard way. chiptool -pendingimfile provides the majority of the information needed, so it's best to reuse this framework. However, after the chipRun entries are defined, they need to be registered into a "supercomputeRun" database entry. My current assumption is that a chip_id or chipRun.label field will be used in the definebyquery statement. The supercomputeRun entry will manage the communication with the remote supercompute resources. This table needs to have the following fields: |
| 262 | | |
| 263 | | * sc_id: main table index |
| 264 | | * stage: current stage being processed for this entry |
| 265 | | * stage_id: reference to the stageRun table |
| 266 | | * state: in addition to the run/full/cleaned values, this will need values pending, auth, and error (described below). |
| 267 | | * last_poll: date string indicating last time this entry was remotely polled. |
| 268 | | * job_id: remote processing job_id. |
| 269 | | * fault: standard fault code |
| 270 | | * path_base: local storage location for supercompute file generation. |
| 271 | | * job_policy: policy state for dealing with partially incomplete runs. |
| 272 | | |
| 273 | | === Processing order === |
| 274 | | |
| 275 | | Once the supercomputeRun is inserted (with state = 'new'), the initial prep task will run, doing the following: |
| 276 | | * Construct list of required files necessary to complete the task on the remote cluster, and their local analogs (if they exist). |
| 277 | | * Construct the command script that defines the resources needed and will launch the srun command. |
| 278 | | * Construct the config script needed for srun to parallelize the processing. |
| 279 | | * Update supercomputeRun to state = 'pending' |
| 280 | | |
| 281 | | With the preparation done, the launch task runs, with the following effects: |
| 282 | | * Check that the connection to the remote supercomputer is available. If not, set the database entry to state = 'auth', update the last_poll entry, and quit. |
| 283 | | * Send list of required files to the remote cluster, and run the validation script on the remote end. This script will produce a file listing any missing entries. This file is returned to the local site, and a transfer list is constructed of the files that need to be pushed to the remote site. |
| 284 | | * Any file pushes that are required are performed, and the command and config scripts are pushed as well. |
| 285 | | * The command script is inserted into the supercomputing queue and the job_id value is obtained. |
| 286 | | * The supercomputeRun is set to state = 'run', and the job_id value updated. |
| 287 | | * A job poll is performed (setting last_poll), and the script either terminates (allowing a future poll to check), or sleeps until the poll determines the job has completed. A future poll will skip the previous steps and return to this point. |
| 288 | | * The remote job status completes, and the resulting files are pulled back from the remote site, including any permanent files (log, stats, SMF, etc), as well as a job_summary compiled at the end of the command script. |
| 289 | | * If the remote job has completed successfully, or if the job_policy is set to "ignore", the local database is updated to reflect that the stageRun has completed successfully, and the state is updated to 'full'. If the job_policy is not set to ignore (job errors are fatal), the local database is updated with information about the jobs that have completed successfully, but the jobs that have failed are marked as so. The supercomputeRun is set to have a fault != 0, and the state = 'error'. This behavior then will require manual intervention to update the database and resolve the failed jobs. |
| 290 | | |
| 291 | | At this point, a master control process will have to step in and launch new supercomputeRun entries for the next step. Upon adding this, the chip->supercomputeRun operation is likely an operation of this master control program operating on a label basis. |
| 292 | | |
| 293 | | === Tasks/scripts/tools === |
| 294 | | |
| 295 | | * Tasks |
| 296 | | * scproc.pro |
| 297 | | * master_control.run |
| 298 | | * prepare.load |
| 299 | | * prepare.run |
| 300 | | * launch.load |
| 301 | | * launch.run |
| 302 | | * launch.poll |
| 303 | | * Scripts |
| 304 | | * sc_prepare_chip.pl |
| 305 | | * sc_prepare_cam.pl |
| 306 | | * sc_prepare_warp.pl |
| 307 | | * sc_prepare_stack.pl |
| 308 | | * sc_validate_files.pl |
| 309 | | * sc_remote_exec.pl |
| 310 | | * sc_validate_processing.pl |
| 311 | | * sc_master_control.pl |
| 312 | | * Tools |
| 313 | | * sctool |
