== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2017.06.19 ===
 * MEH: ippc18 seems have stopped system logging since 6/8  -- /var/log/messages stalled (and ~8GB) but dmesg is reporting recent info, unable to create new screen sessions
  * dmesg now reporting many 
{{{
[4052191.611865] sd 8:0:0:0: [sda] Device not ready: Sense Key : Not Ready [current] 
[4052191.611869] sd 8:0:0:0: [sda] Device not ready: Add. Sense: Logical unit not ready, cause not reportable

[4052215.133126] sd 8:0:0:0: [sda] Device not ready: Sense Key : Not Ready [current] 
[4052215.133133] sd 8:0:0:0: [sda] Device not ready: Add. Sense: Logical unit not ready, cause not reportable
[4052215.133138] end_request: I/O error, dev sda, sector 121605593
[4052215.133187] EXT3-fs error (device sda3): ext3_get_inode_loc: unable to read inode block - inode=3584007, block=7176285

[4052556.736328] Write-error on swap-device (8:0:1870561)
[4052556.808632] sd 8:0:0:0: [sda] Device not ready: Sense Key : Not Ready [current] 
[4052556.808639] sd 8:0:0:0: [sda] Device not ready: Add. Sense: Logical unit not ready, cause not reportable
[4052556.808643] end_request: I/O error, dev sda, sector 1870561
[4052556.808649] Write-error on swap-device (8:0:1870569)
}}}
  * also losing some binary commands like top -- basically non-fuctional now
  * homedir now switched to ippc19 (OLD version from March) -- 
 * MEH: nightly processing check with ippitc (stand alone/isolated homedir on ippc64) -- 
  * nebulous issue since nebdiskd running as ipp and not ippitc -- Gene flipped back
  * ippqub:stdscience_ws -- functioning snapshot from a few days ago rsync'd over old ippc19 version for processing
  * Serge swapped MOPS code needed to run as ippitc in place of ipp 
  * ganglia not available
 * MEH: ipp107-ipp116 mostly full now (actually overly full with ipptopsps loading running) -- regularly past nebulous limit so set repair to allow ipptopsps etc to run -- ipp105,117-120,122 now specifically targeted for ippitc pantasks
  * turning off ipp105 (like ipp117 appears to already be) in nightly processing to avoid overloads -- adding 1x ci0 group to stdscience to help balance 
  * moving WSdiff to start @0600 after nightly mostly finishes to avoid any overloads 
 * MEH: regular cleanup of manually processed and missed products
 * MEH: bzipped ippitc pantasks logs so can make snapshot of that user setup easier/faster
=== Tuesday : YYYY.MM.DD ===

=== Wednesday : 2017.06.21 ===
 * MEH: ippc19.0 homedir filled up last night -- cleared ~2GB of duplicated files staged elsewhere so required nightly processing WSdiff can proceed
=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===
 * 15:30 EAM : restarting pantasks.
 * 20:03 MEH: nightly pantasks not running, unclear why -- no email response so restarting all from scratch @2045
=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===