Changes between Version 4 and Version 5 of Remote_Processing

Timestamp:

Nov 10, 2014, 2:52:02 PM (12 years ago)

Author:

watersc1

Comment:

--

Remote_Processing

@@ +1 @@
+= Remote Processing Maintenance =
+
+== ipplanl scripts ==
+
+ * '''lanl_status.pl'''  Main script to check system status.
+   * {{{lanl_status.pl}}}  Print status information.
+   * {{{lanl_status.pl reconnect}}} Do safe checks of the system, and restart the ssh connection.
+ * '''lanl_control_tool.pl''' Directly start and stop ssh connection.
+   * {{{lanl_control_tool.pl stop}}} Stop ssh connection.
+   * {{{lanl_control_tool.pl start}}} Start ssh connection.
+ * '''lanl_switch_users.pl''' Change the user who is operating the ssh connection/running commands at LANL.
+   * {{{lanl_switch_users.pl USERNAME}}} Switch to user USERNAME
+ * '''revert_stacks.sh''' Revert local stack processing.
+   * {{{revert_stacks.sh}}}
+ * '''drop_bad_data.pl''' Print [PV3_failed_data wiki page] entries and stacktool/warptool commands for runs with unsolvable problems.
+   * {{{drop_bad_data.pl}}} Copy appropriate wiki table entries, and copy and execute the commands listed.
+ * '''revert_lanl.sh''' All purpose "kick" script.  No arguments.  Returns:
+   * Summary of IPP queue, will print correct {{{ln -sf}}} command to promote the next queue file when appropriate.
+   * Summary of LANL queue.  Same checks for new queues.
+   * {{{stat}}} command output of files in ~/.ssh/connection/ to check the age of current ssh connection.
+   * Execute {{{remotetool -revertauth}}} for the LANL label.
+   * Execute {{{remotetool -revertrun}}} for the LANL label.
+   * Construct list of {{{remotetool -revertcomponent}}} commands that need to be copied and run to clear failed sub components.
+   * Generate a ~/lanlstats/mdiag report containing the current /scratch3/ {{{df}}} information and the {{{mdiag}}} processing quota.
+
+== General operation task list ==
+
+ * Check that your username is the active user, using the {{{lanl_switch_users.pl}}} command.
+ * Establish the ssh connection, either with {{{lanl_status.pl reconnect}}} or {{{lanl_control_tool.pl start}}}.
+ * Periodically run the {{{revert_lanl.sh}}} script.  {{{remotetool -revertrun}}} has a cronjob that runs once an hour, which ensures that ssh priority errors don't build up forever, but this is the only way to automatically clear failed components.
+ * Either wait for the ssh connection to be terminated by the remote side, or terminate it manually with {{{lanl_control_tool.pl}}}.  Losing jobs in the middle of a transfer is not fatal, as the {{{rsync}}} commands pick up from the point the connection was lost.
+ * Check {{{drop_bad_data.pl}}} daily, or when obvious warp failures build up.
+
+
 = Setup/authentication =