(Up to PS1 IPP Czar Logs)

Monday : 2012-10-15

• 07:30 Bill: ipp020 is having nfs problems which has clogged stdscience and registration. Fixed the registration problem (we are a couple of hundred exposures behind). Working on ipp020 now.
• 07:50 Restarted stdscience and summitcopy. A stuck job in summitcopy was blocking burntool processing. We have 274 exposures copied but not registered. ipp020 is set to repair and set to off in registration and summit copy.
• 08:00 removed LAP label from stdscience to give nightlyscience all of the horespower
• 11:36 added lap label back. the database replication problem happened again and czartool is behind the times
• 14:14 Bill ran stop slave; set global sql_slave_skip_counter=1; start slave; on ippdb03 and ipp001 to fix the replication problem
• 22:10 MEH: looks like ipp016 has stalling registration and processing.. trying to isolate + clean up
  • restarted registration w/o ipp016 - ok
  • restarted stdscience w/o ipp016 - nightly_science.pl --queue_diffs hanging again.. will probably cause replication error
  • restarted pstamp w/o ipp017 - ok
  • neb-host ipp016 repair -- dont want to reboot tonight if don't have too

Tuesday : 2012-10-16

Wednesday : 2012-10-17

Thursday : 2012-10-18

Friday : 2012-10-19

Saturday : 2012-10-20

Sunday : 2012-10-21