PS1 IPP Czar Logs for the week 2015.02.16 - 2015.02.22

(Up to PS1 IPP Czar Logs)

Monday : 2015.02.16

06:40 EAM: ipp023 down in the night, rebooting now

• 08:50 MEH: OSS diff fault 5 needs quality set to clear

  difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 660733 -skycell_id skycell.1650.047 -dbname gpc1
  

• 10:50 EAM: ippdb06 is up and running as replicant, 2434901 sec behind master. it will probably take several days to catch up. i have purged the ippdb00 binlogs to the entry a few behind the current active binlog for ippdb06. this has freed up 189G, so we can run for a while again.

  mysql> PURGE BINARY LOGS TO 'mysqld-bin.004520';
  

• 23:30 MEH: rolling faults happening on nightly processing from summitcopy through diffs.. ippdb00 connections?

Tuesday : 2015.02.17

• 08:10 EAM: mysql on ippdb06 crashed @ 06:43, restarting it now so it can catch up with ippdb00
• 08:30 MEH: nightly finished, doing storage.hosts.on in stdlocal
• 08:50 EAM: mysql @ ippdb06 up and running, 2271789 sec behind ippdb00. it is going to take > a week to catch up...
• 09:30 EAM: stdlocal is sluggish, stopping for a restart.
• 09:55 EAM: stdlocal up and running.
• 09:55 MEH: QUB stamps slow, restarting pstamp -- with PSPS being used more as well as QUB, this will probably need a near daily restart (really should just do for all pantasks after nightly is finished) -- so doing all other nightly pantasks now
• 10:40 MEH: ipp097 neb-host up, manually adding for nightly processing sum+reg+stdsci
• 10:50 MEH: ipp094 BBU seems bad
• 10:55 MEH: ipp067-ipp082 are ~3.1TB remaining. with normal nightly science was ~4.3TB on 2/12 with normal load of nightly science chip/warp/diffs that get cleaned. Extra data products appear to be going there at ~0.8TB/week.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD