IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 27 and Version 28 of PS1_IPP_Czarlog_20150112


Ignore:
Timestamp:
Jan 13, 2015, 9:47:44 PM (12 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PS1_IPP_Czarlog_20150112

    v27 v28  
    6868 * 17:05 MEH: Gene cleared compute mounts, stsci+pstamp back on. revert update faults caused to ps_ud_QUB
    6969 * 17:25 CZW: stdlanl and stdlocal restarted and running.  At Gene's recommendation, the poll on stdlocal has been decreased to 100 as the default setting until we see how things behave.
     70 * 19:20 MEH: normal processing
     71  * looks like summitcopy ~5-8 behind, registration ~2-3 -- normal 
     72  * WS diffs running seem more load on ipp093,095,090 and may need to go to repair
     73  * waiting for network 10G link to have some regularity -- @1930 -- ~6/5G into/out 6509e w/ poll 100 (roughly that running but warps+camera+stacks also so not clear)
     74 * 19:35 MEH: larger fault spike,  scrambled summitcopy order some, but catching up
     75 * 19:45 MEH: fallout from another massive fault spike -- took about 10 min to recover registration
     76{{{
     77150113 19:32:34InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
     78
     79150113 19:42:30InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
     80}}}
     81 * 20:00 MEH: once things pick back up, load on 10G link is ~7/7-8G.. both ipp/jaws networks are ~2G
     82  * not-targeted ipp092 repair -- to many want to write to node in WT mode
     83  * generally running much better @100 chip in stdlocal 
     84 * 20:30 MEH: stdlocal camera,warp off so chip poll 200 will have a full 200 chips running at a time
     85 * 20:45 MEH: another massive fault wave ~2045 -- 
     86{{{
     87150113 20:47:52InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
     88}}}
     89 * 21:00 MEH: chip poll 200 -- reaching 8-9G on 10G link but...
     90  * more rounds of massive faults -- backlog getting close to shutoff.. may not get good measure..
     91 * 21:15 MEH: is there something else running overloading ippdb00? try stop cleanup to free up connections?
     92 * 20:20 MEH: another huge fault wave -- cleanup still flushing out --
     93
     94
     95
     96
     97
     98
     99
    70100=== Wednesday : YYYY.MM.DD ===
    71101