PS1 IPP Czar Logs for the week 2015.02.16 - 2015.02.22

(Up to PS1 IPP Czar Logs)

Monday : 2015.02.16

06:40 EAM: ipp023 down in the night, rebooting now

• 08:50 MEH: OSS diff fault 5 needs quality set to clear

  difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 660733 -skycell_id skycell.1650.047 -dbname gpc1
  

• 10:50 EAM: ippdb06 is up and running as replicant, 2434901 sec behind master. it will probably take several days to catch up. i have purged the ippdb00 binlogs to the entry a few behind the current active binlog for ippdb06. this has freed up 189G, so we can run for a while again.

  mysql> PURGE BINARY LOGS TO 'mysqld-bin.004520';
  

• 23:30 MEH: rolling faults happening on nightly processing from summitcopy through diffs.. ippdb00 connections?

Tuesday : 2015.02.17

• 08:10 EAM: mysql on ippdb06 crashed @ 06:43, restarting it now so it can catch up with ippdb00
• 08:30 MEH: nightly finished, doing storage.hosts.on in stdlocal
• 08:50 EAM: mysql @ ippdb06 up and running, 2271789 sec behind ippdb00. it is going to take > a week to catch up...
• 09:30 EAM: stdlocal is sluggish, stopping for a restart.
• 09:55 EAM: stdlocal up and running.
• 09:55 MEH: QUB stamps slow, restarting pstamp -- with PSPS being used more as well as QUB, this will probably need a near daily restart (really should just do for all pantasks after nightly is finished) -- so doing all other nightly pantasks now
• 10:40 MEH: ipp097 neb-host up, manually adding for nightly processing sum+reg+stdsci
• 10:50 MEH: ipp094 BBU seems bad
• 10:55 MEH: ipp067-ipp082 are ~3.1TB remaining. with normal nightly science was ~4.3TB on 2/12 with normal load of nightly science chip/warp/diffs that get cleaned. Extra data products appear to be going there at ~0.8TB/week.
• 11:25 MEH: ipp077 seems to be having issues mounting some of the data disks and dmesg log reporting some odd issues with local disk?

  [2015-02-17 07:01:26] journal commit I/O error
  [2015-02-17 07:01:26] __journal_remove_journal_head: freeing b_committed_data
  [2015-02-17 07:01:26] __journal_remove_journal_head: freeing b_committed_data
  [2015-02-17 07:01:26] sd 2:0:0:0: [sdc] Synchronizing SCSI cache
  [2015-02-17 07:01:26] sd 2:0:0:0: [sdc]  
  [2015-02-17 07:01:26] Result: hostbyte=0x04 driverbyte=0x00
  [2015-02-17 07:01:26] sd 2:0:0:0: [sdc] Stopping disk
  [2015-02-17 07:01:26] sd 2:0:0:0: [sdc] START_STOP FAILED
  [2015-02-17 07:01:26] sd 2:0:0:0: [sdc]  
  [2015-02-17 07:01:26] Result: hostbyte=0x04 driverbyte=0x00
  [2015-02-17 07:01:26] Read-error on swap-device (8:32:94345265)
  

• 12:10 MEH: Haydn doing reboot of ipp094 to try and recover BBU -- still bad reporting, he is contacting LSI
• 14:27 CZW: Stopping stdlocal to do a batch lap monitor scan. I'll restart it when I'm done.
• 17:15 MEH: Haydn rebooted ipp077, still doesn't see /dev/sdc but the mounts seem to be fixed now --
• 19:50 MEH: large number of registration and red all over ippmonitor faults..

  Nebulous::Client::move - unhandled fault - database error: error: DBD::mysq
  

  • email to whoever is running extra to stop -- pstamp and cleanup stop to help ease the issue -- may have to stop stdlocal if doesn't clear up
  • stdlocal stop
• 21:00 MEH: been ~20 min w/o fault mess, stdlocal run and see
• 21:15 MEH: massive faults again, stdlocal stop
• 22:15 MEH: been 1hr and no massive fault events -- stdlocal run again
• 22:30 MEH: massive faults -- stdlocal stop
• 22:50 MEH: trying reduced poll in stdlocal -- while observing with the 88in..
• 00:55 MEH: poll up to 200 too high large faults.. back down to 150..

Wednesday : 2015.02.18

• 01:35 MEH: manually quality fault bad warp -- cannot build growth curve (psf model is invalid everywhere)

  warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1477687 -skycell_id skycell.1868.076
  

• 02:15 MEH: pstamp on for past hour or so, issue seems to be driven by the large load of cleanup and stdlocal

• 08:40 MEH: cleanup must be turned back on as soon as nightly finishes in order to catchup from yesterday's faulting issues
• 08:50 MEH: stdlocal poll back to normal
• 09:45 EAM: mysql@ippdb06 crashed, restarted and is now continuing to catch up with ippdb00 -- 2248479 sec behind.
• 09:45 MEH: Gene notes stdlocal needs a restart, stdsci does as well and both will be restarted at at same time, as might as well pstamp

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD