PS1 IPP Czar Logs for the week 2011.03.14 - 2011.03.20

(Up to PS1 IPP Czar Logs)

Monday : 2011.03.14

• czar heather
• 10:44 heather started addstar - no merging
• 10:30 or so Bill fixed some failed warps due to a corrupt camera mask file.
• 11:18 Bill doubled the number of hosts working on distribution since it is so far behind.
• 12:54 reran warp_skycell.pl --warp_id 171166 --skycell_id skycell.0852.098 because one of the output files was corrupted.
• 13:48 heather restarted distribution with 3xdefault hosts

Tuesday : 2011.03.15

• 07:00 Bill lots of red on czartool. Summit copy and registration are far behind. The problem appears to be that nebulous is overloaded and not responding in a timely fashion. Stopped most processing to investigate. Sample error messages

  500 Can't connect to ippdb00.ifa.hawaii.edu:80 (connect: timeout) at /home/panstarrs/ipp/psconfig//ipp-20110218.lin64/lib/Nebulous/Client.pm line 852
  Unable to perform neb-locate: 9 at /home/panstarrs/ipp/psconfig//ipp-20110218.lin64/bin/summit_copy.pl line 233.
  
  
  Unable to stat Nebulous handle neb://ipp030.0/gpc1/20110315/o5635g0395o.310644/o5635g0395o.310644.reg.ota37.log at /home/panstarrs/ipp/psconfig//ipp-20110218.lin64/bin/register_imfile.pl line 61
  

• cleanup has a very large backlog of jobs. It looks like it's in the mode where it takes 10 minutes per (sparse) chipRun. Replication was running as well. With just summit copy, registration, and cleanup running the errors have stopped.
• 07:09 restarting stdscience and distribution. Cleanup is stopped but at this rate will take awhile to empty the queues.
• 7:26 put two repeatedly failing skycells out of their misery.

mysql> update diffSkyfile set fault=0, quality=42 where diff_id = 115074 and skycell_id ='skycell.1455.059';
Query OK, 1 row affected (0.03 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> update diffSkyfile set fault=0, quality=42 where diff_id = 115913 and skycell_id ='skycell.2517.092';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD