PS1 IPP Czar Logs for the week 2014.09.01 - 2014.09.07

(Up to PS1 IPP Czar Logs)

Monday : 2014.09.01

• 14:30 EAM : stsci12 was down per Mark's message -- unresponsive on console. I power-cycled it and it came OK.
• 22:00 EAM : ipp036 was down, no message on console. power cycled it

Tuesday : YYYY.MM.DD

• 10:15 MEH: nightly rate low and net i/o <1G after Gene turned off local pv3 processing, so adding 2x c2 into stdsci (at least to recover systems turned off) --
• 13:10 MEH: nightly not likely to finish before cleanup triggered @2pm, turning off cleanup pantasks until nightly finished so not to bog down the 5 data nodes further
• 14:10 ipp040 being taken down for PS swap -- was set to stop and control reap
  • neb-host down
  • oddly unwant also 5, returned back to 20 when started up again (MEH)
  • w/ ipp040 down, finding missing flats on ippb02

    /data/ippb02.1/nebulous/04/c5/1024845079.gpc1:flatcorr.20100124:GPC1.FLATTEST.305:GPC1.FLATTEST.305.XY15.co.fits
    

  • oddly, normal hosts off are back on in stdsci.. ipp032,060,064.. and newer off ones to beware of crashes.. 047,034,035,036.. including stsciXX -- so maybe better cmd to use instead of this after control reap all?

    control run all
    

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD