== PS1 IPP Czar Logs for the week 2015.02.16 - 2015.02.22 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.02.16 ===
  06:40 EAM: ipp023 down in the night, rebooting now  
 * 08:50 MEH: OSS diff fault 5 needs quality set to clear
{{{
difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 660733 -skycell_id skycell.1650.047 -dbname gpc1
}}}
 * 10:50 EAM: ippdb06 is up and running as replicant, 2434901 sec behind master.  it will probably take several days to catch up.  i have purged the ippdb00 binlogs to the entry a few behind the current active binlog for ippdb06.  this has freed up 189G, so we can run for a while again.
{{{
mysql> PURGE BINARY LOGS TO 'mysqld-bin.004520';
}}}
 * 23:30 MEH: rolling faults happening on nightly processing from summitcopy through diffs.. ippdb00 connections?
=== Tuesday : 2015.02.17 ===
 * 08:10 EAM: mysql on ippdb06 crashed @ 06:43, restarting it now so it can catch up with ippdb00
 * 08:30 MEH: nightly finished, doing storage.hosts.on in stdlocal
 * 08:50 EAM: mysql @ ippdb06 up and running, 2271789 sec behind ippdb00.  it is going to take > a week to catch up...
 * 09:30 EAM: stdlocal is sluggish, stopping for a restart.  
 * 09:55 EAM: stdlocal up and running.
 * 09:55 MEH: QUB stamps slow, restarting pstamp -- with PSPS being used more as well as QUB, this will probably need a near daily restart (really should just do for all pantasks after nightly is finished) -- so doing all other nightly pantasks now
 * 10:40 MEH: ipp097 neb-host up, manually adding for nightly processing sum+reg+stdsci
 * 10:50 MEH: ipp094 BBU seems bad
 * 10:55 MEH: ipp067-ipp082 are ~3.1TB remaining. with normal nightly science was ~4.3TB on 2/12 with normal load of nightly science chip/warp/diffs that get cleaned. Extra data products appear to be going there at ~0.8TB/week. 
 * 11:25 MEH: ipp077 seems to be having issues mounting some of the data disks and dmesg log reporting some odd issues with local disk?
{{{
[2015-02-17 07:01:26] journal commit I/O error
[2015-02-17 07:01:26] __journal_remove_journal_head: freeing b_committed_data
[2015-02-17 07:01:26] __journal_remove_journal_head: freeing b_committed_data
[2015-02-17 07:01:26] sd 2:0:0:0: [sdc] Synchronizing SCSI cache
[2015-02-17 07:01:26] sd 2:0:0:0: [sdc]  
[2015-02-17 07:01:26] Result: hostbyte=0x04 driverbyte=0x00
[2015-02-17 07:01:26] sd 2:0:0:0: [sdc] Stopping disk
[2015-02-17 07:01:26] sd 2:0:0:0: [sdc] START_STOP FAILED
[2015-02-17 07:01:26] sd 2:0:0:0: [sdc]  
[2015-02-17 07:01:26] Result: hostbyte=0x04 driverbyte=0x00
[2015-02-17 07:01:26] Read-error on swap-device (8:32:94345265)
}}}
 * 12:10 MEH: Haydn doing reboot of ipp094 to try and recover BBU -- still bad reporting, he is contacting LSI
 * 14:27 CZW: Stopping stdlocal to do a batch lap monitor scan.  I'll restart it when I'm done.
 * 17:15 MEH: Haydn rebooted ipp077, still doesn't see /dev/sdc but the mounts seem to be fixed now -- 
 * 19:50 MEH: large number of registration and red all over ippmonitor faults.. 
{{{
Nebulous::Client::move - unhandled fault - database error: error: DBD::mysq
}}}
  * email to whoever is running extra to stop -- pstamp and cleanup stop to help ease the issue -- may have to stop stdlocal if doesn't clear up
  * stdlocal stop
 * 21:00 MEH: been ~20 min w/o fault mess, stdlocal run and see
 * 21:15 MEH: massive faults again, stdlocal stop
 * 22:15 MEH: been 1hr and no massive fault events -- stdlocal run again
 * 22:30 MEH: massive faults -- stdlocal stop
 * 22:50 MEH: trying reduced poll in stdlocal -- while observing with the 88in..
 * 00:55 MEH: poll up to 200 too high large faults.. back down to 150..


=== Wednesday : 2015.02.18 ===
 * 01:35 MEH: manually quality fault bad warp -- cannot build growth curve (psf model is invalid everywhere)
{{{
warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1477687 -skycell_id skycell.1868.076
}}}
 * 02:15 MEH: pstamp on for past hour or so, issue seems to be driven by the large load of cleanup and stdlocal

 * 08:40 MEH: cleanup must be turned back on as soon as nightly finishes in order to catchup from yesterday's faulting issues
 * 08:50 MEH: stdlocal poll back to normal
 * 09:45 EAM: mysql@ippdb06 crashed, restarted and is now continuing to catch up with ippdb00 -- 2248479 sec behind.
 * 09:45 MEH: Gene notes stdlocal needs a restart, stdsci does as well and both will be restarted at at same time, as might as well pstamp
 * 10:30 MEH: ipp086,094 still bad BBU, but should be fine with random data (many of the ipp054-066 also have BBU issues being so old and are ok with random data)
 * 11:40 MEH: ippc02 stop/start mysql to rollover the slow.log (>5G) and using excessive space on OS disk
 * 12:30 MEH: older 40T disks in repair (ipp009,011,014,015,017,018,023-031) should be able to be moved to neb-host up -- on the wrong side of the 2x10G link so will need to monitor, should leave in repair any poorer/problematic systems 
  * all should be not targeted except for maybe distribution, ipp010 being used as another alt-data node
  * if notice a problem, high cpu_wait, then back to repair they go
 * 12:45 CZW: stopping stdlocal/stdlanl for the daily lap kick process.
 * 17:30 MEH: update the ipp/nightly processing data host targeting to include ipp097
 * 19:20 MEH: summitcopy+registration massive faults again
  * last was around 19:25:39 -- process list around ~700 (same as in /data/ippc18.0/home/watersc1/monitor_connections.20141210/mon_con.dat5), then drops to 200-400. was this higher processlist while stdlocal slowly cleared the storage node jobs basically?
  * cleanup ~900s, was spike to ~1800s during all the faults 
  * things seem ok now ~20:00
  * restarting all apache servers today may have cleared some problems

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===