== PS1 IPP Czar Logs for the week 2015.06.15 - 2016.06.21 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.06.15 ===
  * 06:19 Bill: stdscience is running sluggishly. I'm going to restart it.
  * 06:37 4 diffs for skycell.0697.027 are failing repeatedly. That stack is bad apparently. Set quality faults for that skycell for diff_id 1161536 1161554 1161573 and 1161061
  * 20:00 EAM: restarting ipp pantasks for smoother nightly processing.

=== Tuesday : 2015.06.16 ===
 * 07:40 MEH: some WS diffs are stalled from primary neb-instance of stack being on ippb06, i.e.,
{{{
1162409 	skycell.1480.080 	ThreePi.WS.nightlyscience 	ThreePi.20150616
neb://any/gpc1/LAP.ThreePi.20130717/2014/02/15/RINGS.V3/skycell.1480.080/RINGS.V3.skycell.1480.080.stk.3112920.unconv.fits
}}}
 * 07:50 MEH: ipp077, 080 ganglia apparently haven't been reporting since early May...
{{{
/etc/init.d/gmond restart
}}}
 * 11:50 MEH: two very old gpc2 exposures just repeatedly failing in registration for some bad? ota -- setting to state drop in newExp, if wanted can deal with later
{{{
update newExp set state="drop" where exp_id=1806;
update newExp set state="drop" where exp_id=1021;
}}} 
 * 17:45 CZW: Investigating a sudden string of failures on the phase 2 shuffle indicates that ipp066 isn't automounting anything that isn't already mounted.
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===