PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2013.04.01

mark is czar

• 08:05 MEH: remaining few nightly science exposures finishing up
• 09:38 Bill: ippc03 is running out of space in /tmp. The nebulous_server.log file had grown to 22G. Stopped apache, moved the file out of the way, restarted apache, then deleted it.
• 11:10 MEH: ipp05,08,09 nebulous_sever.log >22G as well, leaving ~4G on / OS disk and should be cycled out as well -- done. ippc07 will be next, only ~9G left available
• 19:00 MEH: MD09 deepstacks done so compute3 back to stdsci and stack until start MD08 refstack ~tomorrow
• 22:20 MEH: appear to have lost connection from ifa to production cluster? ganglia also reporting all systems down..

Tuesday : 2013.04.02

mark is czar

• 07:00 MEH: network issue with summit, but also something odd with production cluster and looks like some systems are down and some were rebooted.
  • still down: 028,033,036,046,048,c17,c19,db02 and console not reachable -- was ganglia problem for some, ipp046,048,c17,c19 actually down/off network, cannot connect via console
  • checking if the stsci nodes were power cycled -- no, so probably good thing
  • ippdb00,01,03 have been so may need to check mysql
  • various data (non-stsci data) nodes have been rebooted it seems
• 08:30 MEH: restarted screens and czarpoll on ippc11, will wait for roboczar until systems back running
• 08:40 MEH: rebooted ipp036 to fix mount issues, didn't come back up cleanly after whatever happened last night. need to scan mounts on all machines
  • serge put ipp036 into neb-host repair -- need to put back up - ok
  • ipp028,033,036,060 disks aren't always mounting -- Gene noted ypbind gone rogue, need to kill and restart with nfs (sudo tcsh, su - otherwise once kill ypbind no sudo access of course)
  • some wave1 machines trouble mounting wave4
• 09:30 MEH: ganglia is showing several machines high 1-m load but nothing running, gmond probably needs to be restarted on most/all machines -- done, and login seems fine except for ipp041 date below
• 09:40: MEH: ippc17 and ippc19 down, ippRequestServer (pss, datastore db and replication) so not possible for PSS to run until fixed
• 10:00 MEH: ipp041 date is wrong -- 10:25 manually set to w/in 1s. how to fix time sync?
• 10:20 Bill added PSS MOTD, to remove -- cp /data/ippc17.0/pstamp/web/pstamp_motd.html.empty /data/ippc17.0/pstamp/web/pstamp_motd.html
• 11:45 MEH: ipp048 and ippc19 still problems, ipp048 to down. -- Haydn and Rita having trouble logging into ipp048 console, I can log in and disk scan taking the time for reboot -- back up now, neb-host up.
  • ippc19 still being investigated -- burning smell, out for the time being
• 12:05 MEH: datanodes back online, mounts appear fine. slowly starting pantasks -- summitcopy, registration, stdscience, stack
• 12:20 MEH: nebdiskd needs to be started on ippdb00
• 12:35 Serge fixed crashed czardb table processed -- repair table processed. loaded_to_datastore also crashed and repaired. eventually the czarpoll page will be uptodate. loaded_to_ODM also crash and repaired.
• 12:50 MEH: distribution, publishing, detrend started. pstamp and update will wait until decide on ippc17 replication
• 13:40 MEH: finding other machines with close but wrong time by hours.. more importantly ippdb00,01 are off.. so all processing to stop
• 15:00 MEH: system times fixed and /etc/init.d/ntpd restart, ippdb00,01 mysql restarted, processing restarted
• 15:05 MEH: wave1 need reboot with testing 3.7.6 kernel or may have disk issues with all up in nebulous -- all into repair and removed from processing, slowly rebooting with 3.7.6 kernel
• 15:15 Bill noticed pstamp confused with ports from detrend. fixed and pstamp running again
• 15:18 Bill removed the "pstamp is down" message from postage stamp web pages.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD