IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 3 and Version 4 of PS1_IPP_Czarlog_20161205


Ignore:
Timestamp:
Dec 7, 2016, 9:24:32 PM (10 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PS1_IPP_Czarlog_20161205

    v3 v4  
    99
    1010=== Wednesday : 2016.12.07 ===
    11 
     11 * MEH: ipp118-122.x were not completely setup and/or checked for ipp use, link for /local/ipp broken -- fixed, adding as group s7, adding to nightly science targeting and heavily loading CPU and IO with MOPS/QUB processing until weather clears
     12 * MEH: ippcXX nodes lately have been having NFS hangs that stall MOPS pstamp request (and QUB if there had been any data) -- restarting nfs/rpc.statd seems to clear -- czar needs to keep eye on
     13 * MEH: stamps/updates for LAP.PV3 showing ongoing faults due missing files from stsci shuffle -- some reporting ipp118 but looks like due to IP being reused
     14 * MEH: gpc1 dumps to ipp001 have been 0 size for past couple days it appears due to /data/ipp001.0 being full -- adding an email warning crontab on ipp001, running cleanup script and rm'ing incomplete dumps so not to contaminate archive...
     15  * will set one up in ipp@ippc18 to also monitor homedir
     16 * MEH: Haydn replace ipp122 10G card and wanted to replace ipp083 BBU -- out of processing and neb-host down
     17  * ipp122 back up but cannot log into while disks mount ok -- leave out of processing and in repair -- then later was able to log into again
     18  * ipp083 unclear, leave in repair until morning
    1219=== Thursday : 2016.12.08 ===
    1320