PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : 2017.09.14

• MEH: ippc65 down since early AM, Hadyn power cycled
• MEH: ipp@ippc19 crontab adding /export/ippdbXX check added now since ippdb08 filled up surprisingly today
• MEH: ippdb06,db08 ganglia plots missing since weekend.. -- critical machines, useful to have plots.. -- /etc/init.d/gmond restart
• MEH: sending missed and misc nightly processing to cleanup again
• 15:30 EAM : ran out of space on ippdb08 /export/ippdb08.0 due to a problem with the binlogs. I cleared out some old files first (a test dump and an old, partial copy of nebulous mysql). I then discovered that the binlogs had been getting very large for a while: this was caused by an error introduced into gzgetexp on July 15 which resulting it in attempting to re-download every failed exposure, instead of only the past night. I fixed the code in pzgetexp to only check for the past 100 exposures. I also cleared the binlogs up to Sept 12.
• 16:20 EAM : updated ippMonitor to report replication status for ippdb01 for nebulous (instead of ipp115)
• MEH: ippc61 appears to have been crashed since weekend? cannot ssh into, no console login (or messages) -- power cycle twice without booting -- email Haydn to look at if time -- Haydn thinks its already on his list of ones to work on and hasn't been up for a while and maybe IP address mixup -- conflicts w/ what ganglia was reporting and that Weryk has been actively using it as well
• MEH: nightly pantasks stop.block.restart seems to get pstamp into weird/broken state if trying to make stamps/update with faults -- happened a couple times now -- manually clearing stalled MOPS jobs...

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD