== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : YYYY.MM.DD ===
  
=== Tuesday : YYYY.MM.DD ===

=== Wednesday : 2017.05.24 ===
 * MEH: adding additional cleanup attempts from 2016 that had failed during the shuffle/move so that MOPS+QUB can get more of the stamps they need
 * 16:00 CZW: The migration to the updated czartool (r40008) with summit, download, new, and raw Exp stage handlers is complete.  This update also ensures that temp plot files are cleaned.
=== Thursday : 2017.05.25 ===
 * MEH: ipps11 appears to have been down/unresponsive on ganglia since yesterday ~noon -- barely log into, io error on commands
{{{
-- multiple entries on console
[252083.738963] Uhhuh. NMI received for unknown reason 3d on CPU 0.
[252083.745002] Do you have a strange power saving mode enabled?
[252083.750774] Dazed and confused, but trying to continue
[252086.206723] Uhhuh. NMI received for unknown reason 2d on CPU 0.
[252086.212754] Do you have a strange power saving mode enabled?
[252086.218526] Dazed and confused, but trying to continue
-- after power cycle boot -- odd message, if it matters
 * Starting local .../etc/conf.d/local.start: line 13: /sys/devices/system/cpu/sched_mc_power_savings: No such file or directory
-- seems ok for now
}}}
=== Friday : 2017.05.26 ===
 * MEH: ipp121.1 not mountable, setting neb-host down --volume -- probably needs xfs_check run
{{{
[1486417.010902] XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1607 of file fs/xfs/xfs_alloc.c.  Caller 0xffffffff8121ecf9
...
[1486417.010970] XFS (sdb1): xfs_do_force_shutdown(0x8) called from line 3734 of file fs/xfs/xfs_bmap.c.  Return address = 0xffffffff8122a4e7
[1486417.011119] XFS (sdb1): Corruption of in-memory data detected.  Shutting down filesystem
[1486417.011121] XFS (sdb1): Please umount the filesystem and rectify the problem(s)
...
[1518433.309029] XFS (sdb1): xfs_log_force: error 5 returned.
}}}
 * 13:30 EAM : Haydn rebooted and cleared the xfs log for ipp121.1.  It is behaving OK now, so I've put it in repair.
 * 16:55 EAM : restarting pantasks

=== Saturday : YYYY.MM.DD ===

=== Sunday : 2017.05.28 ===
 * 12:00 EAM : restarting pantasks