IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 8 and Version 9 of PS1_IPP_Czarlog_20140707


Ignore:
Timestamp:
Jul 8, 2014, 1:48:00 PM (12 years ago)
Author:
eugene
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PS1_IPP_Czarlog_20140707

    v8 v9  
    1616
    1717  * 07:50 EAM : the chiller at MRTC-B failed in the night and Haydn had to shut down the whole cluster.  machines are slowly coming back up now. 
     18  * 13:00 EAM : gavin brought back up the machines which could be power-cycled from Manoa. ippdb00 had trouble starting mysql : it needed to recover from the binlogs, and this was slow, but eventually succeeded.  jaws machines needed to be manually booted by Haydn with some difficulty.  ipp030 also gave some trouble.  I initially proposed that we move ippdb01 as well today, but it took so long for Haydn to get past security that we gave up on that plan for today.  At this point, processing is running, but there are lots of fault, probably due to block nebulous files.  I need to clear these out before we will be in reasonable shape.
    1819
    1920  * 13:45 EAM : ipp059 just crashed: wiki:ipp059-20140708-crash.log