== PS1 IPP Czar Logs for the week 2015.04.06 - 2015.04.13 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.04.06 ===
 * 15:30 MEH: starting the regular restart of nightly pantasks
 * 15:40 MEH: ippc0x apache servers low on disk space, restarting apache and purging old log
 * 15:50 Haydn reports ipp017 back up, put into neb-host repair
  * clearing old and then stalled WS diff distribution -- files got cleaned again.. so update again
 * 17:30 MEH: ipp088,090,093,095,097 all in repair -- put to neb-host up and see if past getting slammed since earlier storage nodes are red..
 * 19:50 MEH: slow processing, nightly nodes still on in PV3 diff
 * 20:50 MEH: reg fault backing up processing?
 * 22:55 MEH: killed long running thread db00

=== Tuesday : 2015.04.07 ===
 * 09:25 EAM : today, we are changing the nebulous database mysql master from ippdb00 (nearly full disk) to ippdb08 (new hardware).  Here are my notes on the process:
  * 09:25 EAM : stopping the various pantasks (~ipp/*, ~ipptest/pv3diff*, ~ipplanl)
  * 09:50 EAM : everything is shut down, stopping apache on ippc17 and ippc30 (pstamp and datastore)
  * 10:00 EAM : shutdown all apache servers on ippc01 - ippc10
   * everything is quiet on ippdb00 : no queries except the slaves.  after excuting 'flush tables with read lock' on ippdb00, the master status is:
{{{
mysql> SHOW MASTER STATUS;
+-------------------+----------+--------------+------------------+-------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| mysqld-bin.005160 | 80910016 |              |                  |                   |
+-------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
}}}
  * 10:04 EAM : shutting down the mysql engine on ippdb00, ippdb02, ippdb06, ippdb08
  * 10:55 EAM : set the new master configuration for both ippdb02 and ippdb06.  started the slaves and tested (inserted a row into the unused table 'log' and then deleted it).  the slaves see the changes on the master.  I had some confusion setting up the replication users for ippdb06 and ippdb02 on ippdb08 (I needed to use the following command):
{{{
GRANT REPLICATION SLAVE ON *.* to 'USER'@'ippdb02.ifa.hawaii.edu' identified by 'PASSWORD';
}}}
  * 11:05 EAM : nebulous.ipp now points at ippdb08.  I started up the ippc01 - ippc10 apache servers and all seems well.
  * 11:12 EAM : started apache on ippc17 and ippc30.  started nebdiskd.
  * 11:36 EAM : pv3diff and pv3diffleft started
  * 11:40 EAM : ~ipp standard pantasks started
   * MEH: ipp nightly processing targeting modified, please do not rebuild the ops tag -- merged now
  * 12:55 EAM : i've launched the gpc2 MOPS and photometry test images and added gpc2 to the stdscience pantasks (also added it to stdscience/input so it will be available)
 * 14:10 MEH: something is trying to heavily write to ipp082.. wonder if this is partial cause -- into repair until behaving better..
{{{
[2015-04-07 13:28:08] 3w-sas: scsi0: AEN: INFO (0x04:0x0029): Verify started:unit=0.
}}}
 * 14:45 MEH: ipp090 having CMCI storm spamming logs -- putting into neb-host repair until decide what to do about it
 * 15:15 MEH: ipp082 was almost ok for ~1hr and something is again trying to kill it.... before could put to repair looks to have corrected itself -- aligns with start of gpc2 diffs?
 * 16:30 Haydn rebooting ipp090 to try and fix the CMCI messages -- took a second reboot to get back online. leave in repair for tonight
 * 19:50 MEH: stsci10 appears unresponsive and cannot log into -- neb-host down
 * 20:20 EAM: I've power cycled it.  it seems to be booting ok.
 * 00:25 MEH: looks like stsci10 is back up but still neb-host down.. raising to repair for the night
=== Wednesday : 2015.04.08 ===
 * 11:00 MEH: ipp090 seems okay overnight, so back to neb-host up
 * 11:30 EAM: stopping and restarting pv3diff and pv3diffleft
 * 14:30 EAM: i was running a parallel dvo merge on stsci19 and it seems to have crashed.  i'm not sure if there was a causal connection, but in any event i've power cycled it.

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===