== PS1 IPP Czar Logs for the week 2014-03-03 - 2014-03-09 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : YYYY.MM.DD ===
  * 16:19 Bill ran add.sts.rp macro in stdscience to start up STS processing
=== Tuesday : 2014.03.04 ===
mark is czar

 * 06xx Bill, Gene rebooted ippdb01 and got nightly going again
 * 06:50 MEH: ippdb01 down again, looks like root rebooted ~0700 -- looks like rebooted both times possibly missing a little RAM compared to earlier in week on ganglia
  * STS label out of stdsci until night finished.
 * 07:20 MEH: czarpoll and roboczar stopped on ippc11 -- restarted
 * 07:30 MEH: staticsky was stopped w/o a note, set back to run.. STS label back into stdsci
 * 10:10 MEH: older STS throwing many fault 2 burntool issue -- chip.revert.off..
{{{
burntool state vs burntoolStateGood : -13 vs 14
Image burntool version does not match current accepted version. at /home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/chip_imfile.pl line 830
	main::my_die('Image burntool version does not match current accepted version.', 89358, 953683, 'XY01', 2) called at /home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/chip_imfile.pl line 328
}}}
  * Bill has been changing label for these to STS.rp.2013.hold
 * 10:20 MEH: ippc18.0 <70G.. doing log archive will free up ~30G only. apache node /tmp space >20G and okay.
 * 11:50 MEH: noticed ipp004 has been constantly faulting stacks, was missing /local/ipp/tmp...
 * 13:20 MEH: ipp040,043,050,052,053 being continual slugs on chip processing, 2-3ks (as has been noted many times in past) -- neb-host repair 
 * 13:30 MEH: stdsci >>100k Njobs and becoming somewhat unresponsive, will do necessary regular restart shortly before nightly
 * 14:25 MEH: ippc08 is down -- nothing on console
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

 * sometime -- Rita to do IPP port swap from 48 port switch to ippcore slot 9
=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===