IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 7 and Version 8 of PS1_IPP_Czarlog_20161128


Ignore:
Timestamp:
Dec 1, 2016, 12:12:44 PM (10 years ago)
Author:
watersc1
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PS1_IPP_Czarlog_20161128

    v7 v8  
    2929
    3030  * 11:38 HAF No network from 2:25am to about 11:00am.  Pantasks in a funky state, restarting
     31  * 12:06 CZW: The new MRTCB nebulous nodes should be fine to accept data.  Setting to up.
     32  * 12:10 CZW: I'm increasing the loading of the ipptest/replication pantasks to speed up the MRTCB->ITC raw shuffle.  This increase will use 6 times loading of the x2 and x3 nodes.  The notes for this process are listed [wiki:ITC_shuffle on the wiki here].  It would be helpful if czars can check on this pantasks every day or so, and try to kill hung jobs.  The standard execution time is about 400-600s, so jobs with times in the 6000+s range are probably having mount issues.  The wiki page discusses the ways I've been dealing with this (kill the jobs, restart nfs if needed to clear bad mounts).  Although it's best to have things complete, failed jobs can be easily retried. 
    3133=== Friday : 2016.12.02 ===
    3234