PS1 IPP Czar Logs for the week 2014.09.01 - 2014.09.07

(Up to PS1 IPP Czar Logs)

Monday : 2014.09.01

• 14:30 EAM : stsci12 was down per Mark's message -- unresponsive on console. I power-cycled it and it came OK.
• 22:00 EAM : ipp036 was down, no message on console. power cycled it

Tuesday : 2014.09.02

• 10:15 MEH: nightly rate low and net i/o <1G after Gene turned off local pv3 processing, so adding 2x c2 into stdsci (at least to recover systems turned off) --
• 13:10 MEH: nightly not likely to finish before cleanup triggered @2pm, turning off cleanup pantasks until nightly finished so not to bog down the 5 data nodes further
• 14:10 ipp040 being taken down for PS swap -- Gene set to stop and control reap
  • MEH: neb-host down
  • MEH: oddly unwant also 5, returned back to 20 when started up again
  • MEH: w/ ipp040 down, finding missing flats on ippb02

    /data/ippb02.1/nebulous/04/c5/1024845079.gpc1:flatcorr.20100124:GPC1.FLATTEST.305:GPC1.FLATTEST.305.XY15.co.fits
    

  • MEH: oddly, normal hosts off are back on in stdsci.. ipp032,060,064.. and newer off ones to beware of crashes.. 047,034,035,036.. including stsciXX -- so maybe better cmd to use instead of this after control reap all? -- need to do normal restart of stsci instead?

    control run all
    

• 15:15 MEH: stdsci behaving badly, books full of junk.. restarting..
• 17:15 MEH: summitcopy all PENDING.. had reap done? not logged.. so just restarting to start the nightly downloads
• 18:30 MEH: suspect possible problem/unbalanced issue w/ ipp071 -- putting into repair seems to improve processing, so may leave out for some of nightly and see
• 19:10 MEH: nightly finally finished, diffims seemed to have been jumbled and many published towards end. putting in the WS labels until tonight's nightly starts, turning cleanup on to watch how things go with ipp071 in repair.
  • generally higher loads on ipp067-ipp070 when ipp071 in repair, though ipp069 more so as it has just slightly less than ipp070 so more is put there? in any case seems to be able to handle it better. maybe 2-3TB should be shuffled to ipp071 and see if that helps?
    • have ~2.4TB of MD stack tests on ipp060 that should be moved, so will try that
    • or maybe just leave in repair and start rsync of ipp047?
    • or maybe 0.5-1TB off the stsci nodes for addstar space?
• 20:00 MEH: will try PV3.ipp processing back on until nightly starts to see how that goes

Wednesday : 2014.09.03

• 07:10 MEH: ipp047 didn't make it very long, down ~6am
• 08:50 MEH: restarted NFS on ipp071, neb-host up to see if flushed a problem -- ipplanl/stdlocal pantasks well past needing restart and underloaded, so not a good test
• 10:10 MEH: pstamp running for a while, restarting
• 13:55 MEH: possible update problems expected if files on disks w/ no space (e.g., stsci11.1) --

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD