PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2017.06.19

• MEH: ippc18 seems have stopped system logging since 6/8 -- /var/log/messages stalled (and ~8GB) but dmesg is reporting recent info, unable to create new screen sessions
  • dmesg now reporting many

    [4052191.611865] sd 8:0:0:0: [sda] Device not ready: Sense Key : Not Ready [current] 
    [4052191.611869] sd 8:0:0:0: [sda] Device not ready: Add. Sense: Logical unit not ready, cause not reportable
    
    [4052215.133126] sd 8:0:0:0: [sda] Device not ready: Sense Key : Not Ready [current] 
    [4052215.133133] sd 8:0:0:0: [sda] Device not ready: Add. Sense: Logical unit not ready, cause not reportable
    [4052215.133138] end_request: I/O error, dev sda, sector 121605593
    [4052215.133187] EXT3-fs error (device sda3): ext3_get_inode_loc: unable to read inode block - inode=3584007, block=7176285
    
    [4052556.736328] Write-error on swap-device (8:0:1870561)
    [4052556.808632] sd 8:0:0:0: [sda] Device not ready: Sense Key : Not Ready [current] 
    [4052556.808639] sd 8:0:0:0: [sda] Device not ready: Add. Sense: Logical unit not ready, cause not reportable
    [4052556.808643] end_request: I/O error, dev sda, sector 1870561
    [4052556.808649] Write-error on swap-device (8:0:1870569)
    

  • also losing some binary commands like top -- basically non-fuctional now
  • homedir now switched to ippc19 (OLD version from March) --
• MEH: nightly processing check with ippitc (stand alone/isolated homedir on ippc64) --
  • nebulous issue since nebdiskd running as ipp and not ippitc -- Gene flipped back
  • ippqub:stdscience_ws -- functioning snapshot from a few days ago rsync'd over old ippc19 version for processing
  • Serge swapped MOPS code needed to run as ippitc in place of ipp
  • ganglia not available
• MEH: ipp107-ipp116 mostly full now (actually overly full with ipptopsps loading running) -- regularly past nebulous limit so set repair to allow ipptopsps etc to run -- ipp105,117-120,122 now specifically targeted for ippitc pantasks
  • turning off ipp105 (like ipp117 appears to already be) in nightly processing to avoid overloads -- adding 1x ci0 group to stdscience to help balance
  • moving WSdiff to start @0600 after nightly mostly finishes to avoid any overloads
• MEH: regular cleanup of manually processed and missed products
• MEH: bzipped ippitc pantasks logs so can make snapshot of that user setup easier/faster

Tuesday : YYYY.MM.DD

Wednesday : 2017.06.21

• MEH: ippc19.0 homedir filled up last night -- cleared ~2GB of duplicated files staged elsewhere so required nightly processing WSdiff can proceed

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

• 15:30 EAM : restarting pantasks.
• 20:03 MEH: nightly pantasks not running, unclear why -- no email response so restarting all from scratch @2045

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD