PS1 IPP Czar Logs for the week 2015.06.15 - 2016.06.21

(Up to PS1 IPP Czar Logs)

Monday : 2015.06.15

• 06:19 Bill: stdscience is running sluggishly. I'm going to restart it.
• 06:37 4 diffs for skycell.0697.027 are failing repeatedly. That stack is bad apparently. Set quality faults for that skycell for diff_id 1161536 1161554 1161573 and 1161061
• 20:00 EAM: restarting ipp pantasks for smoother nightly processing.

Tuesday : 2015.06.16

• 07:40 MEH: some WS diffs are stalled from primary neb-instance of stack being on ippb06, i.e.,

  1162409 	skycell.1480.080 	ThreePi.WS.nightlyscience 	ThreePi.20150616
  neb://any/gpc1/LAP.ThreePi.20130717/2014/02/15/RINGS.V3/skycell.1480.080/RINGS.V3.skycell.1480.080.stk.3112920.unconv.fits
  

• 07:50 MEH: ipp077, 080 ganglia apparently haven't been reporting since early May...

  /etc/init.d/gmond restart
  

• 11:50 MEH: two very old gpc2 exposures just repeatedly failing in registration for some bad? ota -- setting to state drop in newExp, if wanted can deal with later

  update newExp set state="drop" where exp_id=1806;
  update newExp set state="drop" where exp_id=1021;
  

• 17:45 CZW: Investigating a sudden string of failures on the phase 2 shuffle indicates that ipp066 isn't automounting anything that isn't already mounted.
  • Gene fixed this: /etc/init.d/autofs zap && /etc/init.d/autofs start
• 18:00 MEH: restarting registration to incorporate longstanding failing in revert exp and test reducing the exec time since @1800s it takes ~2hrs to get back to gpc1 and MOPS processing can get quite a ways behind by then
• 21:50 MEH: ipp067 very high load and processing rate very low with many errors.. putting into repair while looking at errors

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD