PS1 IPP Czar Logs for the week 2015.04.06 - 2015.04.13

(Up to PS1 IPP Czar Logs)

Monday : 2015.04.06

• 15:30 MEH: starting the regular restart of nightly pantasks
• 15:40 MEH: ippc0x apache servers low on disk space, restarting apache and purging old log
• 15:50 Haydn reports ipp017 back up, put into neb-host repair
  • clearing old and then stalled WS diff distribution -- files got cleaned again.. so update again
• 17:30 MEH: ipp088,090,093,095,097 all in repair -- put to neb-host up and see if past getting slammed since earlier storage nodes are red..
• 19:50 MEH: slow processing, nightly nodes still on in PV3 diff
• 20:50 MEH: reg fault backing up processing?
• 22:55 MEH: killed long running thread db00

Tuesday : 2015.04.07

• 09:25 EAM : today, we are changing the nebulous database mysql master from ippdb00 (nearly full disk) to ippdb08 (new hardware). Here are my notes on the process:
  • 09:25 EAM : stopping the various pantasks (~ipp/*, ~ipptest/pv3diff*, ~ipplanl)
  • 09:50 EAM : everything is shut down, stopping apache on ippc17 and ippc30 (pstamp and datastore)
  • 10:00 EAM : shutdown all apache servers on ippc01 - ippc10
    • everything is quiet on ippdb00 : no queries except the slaves. after excuting 'flush tables with read lock' on ippdb00, the master status is:

      mysql> SHOW MASTER STATUS;
      +-------------------+----------+--------------+------------------+-------------------+
      | File              | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
      +-------------------+----------+--------------+------------------+-------------------+
      | mysqld-bin.005160 | 80910016 |              |                  |                   |
      +-------------------+----------+--------------+------------------+-------------------+
      1 row in set (0.00 sec)
      

  • 10:04 EAM : shutting down the mysql engine on ippdb00, ippdb02, ippdb06, ippdb08
  • 10:55 EAM : set the new master configuration for both ippdb02 and ippdb06. started the slaves and tested (inserted a row into the unused table 'log' and then deleted it). the slaves see the changes on the master. I had some confusion setting up the replication users for ippdb06 and ippdb02 on ippdb08 (I needed to use the following command):

    GRANT REPLICATION SLAVE ON *.* to 'USER'@'ippdb02.ifa.hawaii.edu' identified by 'PASSWORD';
    

  • 11:05 EAM : nebulous.ipp now points at ippdb08. I started up the ippc01 - ippc10 apache servers and all seems well.
  • 11:12 EAM : started apache on ippc17 and ippc30. started nebdiskd.
  • 11:36 EAM : pv3diff and pv3diffleft started
  • 11:40 EAM : ~ipp standard pantasks started
    • MEH: ipp nightly processing targeting modified, please do not rebuild the ops tag -- merged now
  • 12:55 EAM : i've launched the gpc2 MOPS and photometry test images and added gpc2 to the stdscience pantasks (also added it to stdscience/input so it will be available)
• 14:10 MEH: something is trying to heavily write to ipp082.. wonder if this is partial cause -- into repair until behaving better..

  [2015-04-07 13:28:08] 3w-sas: scsi0: AEN: INFO (0x04:0x0029): Verify started:unit=0.
  

• 14:45 MEH: ipp090 having CMCI storm spamming logs -- putting into neb-host repair until decide what to do about it
• 15:15 MEH: ipp082 was almost ok for ~1hr and something is again trying to kill it.... before could put to repair looks to have corrected itself -- aligns with start of gpc2 diffs?
• 16:30 Haydn rebooting ipp090 to try and fix the CMCI messages -- took a second reboot to get back online. leave in repair for tonight
• 19:50 MEH: stsci10 appears unresponsive and cannot log into -- neb-host down
• 20:20 EAM: I've power cycled it. it seems to be booting ok.
• 00:25 MEH: looks like stsci10 is back up but still neb-host down.. raising to repair for the night

Wednesday : 2015.04.08

• 11:00 MEH: ipp090 seems okay overnight, so back to neb-host up

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD