PS1 IPP Czar Logs for the week 2011.03.14 - 2011.03.20

(Up to PS1 IPP Czar Logs)

Monday : 2011.03.14

• czar heather
• 10:44 heather started addstar - no merging
• 10:30 or so Bill fixed some failed warps due to a corrupt camera mask file.
• 11:18 Bill doubled the number of hosts working on distribution since it is so far behind.
• 12:54 reran warp_skycell.pl --warp_id 171166 --skycell_id skycell.0852.098 because one of the output files was corrupted.
• 13:48 heather restarted distribution with 3xdefault hosts

Tuesday : 2011.03.15

• 07:00 Bill lots of red on czartool. Summit copy and registration are far behind. The problem appears to be that nebulous is overloaded and not responding in a timely fashion. Stopped most processing to investigate. Sample error messages

  500 Can't connect to ippdb00.ifa.hawaii.edu:80 (connect: timeout) at /home/panstarrs/ipp/psconfig//ipp-20110218.lin64/lib/Nebulous/Client.pm line 852
  Unable to perform neb-locate: 9 at /home/panstarrs/ipp/psconfig//ipp-20110218.lin64/bin/summit_copy.pl line 233.
  
  
  Unable to stat Nebulous handle neb://ipp030.0/gpc1/20110315/o5635g0395o.310644/o5635g0395o.310644.reg.ota37.log at /home/panstarrs/ipp/psconfig//ipp-20110218.lin64/bin/register_imfile.pl line 61
  

• cleanup has a very large backlog of jobs. It looks like it's in the mode where it takes 10 minutes per (sparse) chipRun. Replication was running as well. With just summit copy, registration, and cleanup running the errors have stopped.
• 07:09 restarting stdscience and distribution. Cleanup is stopped but at this rate will take awhile to empty the queues.
• 7:26 put two repeatedly failing skycells out of their misery.

  mysql> update diffSkyfile set fault=0, quality=42 where diff_id = 115074 and skycell_id ='skycell.1455.059';
  Query OK, 1 row affected (0.03 sec)
  Rows matched: 1  Changed: 1  Warnings: 0
  
  mysql> update diffSkyfile set fault=0, quality=42 where diff_id = 115913 and skycell_id ='skycell.2517.092';
  Query OK, 1 row affected (0.00 sec)
  Rows matched: 1  Changed: 1  Warnings: 0
  

• 08:21 turning replication back on
• 08:43 found 103 magicDSRuns in state failed_revert. Set them to new. Turned off destreak to see if the faults repeat.
• 08:52 all of the faults reverted successfully. revert off destreak on
• 09:24 destreak revert back on.
• 09:40 cleanup set to run with poll limit of 4
• 11:30 heather reverted 4 skycells for MD08.jtrp and added them back into stdsci - 3 are fault 2 (related to various ipps being down/rebooted lately). one is a 'fix this code'. Once these fail (or success) heather is going to send them to the cleaners.
• 11:40 heather set MD08.jtrp to be cleaned. the 3 fault 2s succeeded and the fault4 faulted again.
• 11:45 All processing is stopped
• 12:18 nebulous replication Parameters:

  mysql> SHOW MASTER STATUS\G
  *************************** 1. row ***************************
              File: mysqld-bin.002221
          Position: 1068308831
      Binlog_Do_DB: 
  Binlog_Ignore_DB: 
  

Dump is on ippdb00:

mysqldump --master-data nebulous -u root > /export/ippdb00.0/ipp/nebulous/nebulous_dump-20110315T121800.sql

mysqldump took TODO hours

• 13:24 Finally got the lock on ippdb01 (1 hour to complete. Weird)

  mysql> SHOW MASTER STATUS;
  | File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
  | mysqld-bin.018618 |  4201327 |              |                  | 
  

For information: Beginning: 2011-03-15 13:24:40; End: 2011-03-15 13:58:17; Size of dump: 49GB (52416179007)

Wednesday : 2011-03-16

Thursday : 2011-03-17

Friday : 2011-03-18

Saturday : 2011-03-19

Sunday : 2011-03-20