PS1 IPP Czar Logs for the week 2013.03.11 - 2013.03.17

(Up to PS1 IPP Czar Logs)

Monday : 2013.03.11

• 09:40 Bill noticed that two distRuns from the other night had not completed. The problem was a missing output file (subkernel) from the input diffRun. The instance was in nebulous but the file on stsci07 was not. Reran the component using tools/rerundiffskyfile.pl. This worked
• 13:06 Bill set all pantasks to stop in order to rebuild ippTools with a change to pstampRequest
• 13:35 Bill: Restarted stdscience and set other pantasks to run
• 13:50 Bill: regenerated two lost burntool tables and recovered o5518g0719o.ota26.fits

Tuesday : 2013.03.12

• 15:11 Bill: I am running new task and script to process lapGroups which are collections of lapRuns that have "finished" and have each of the 5 filters. staticskyruns for all filter combinations

Wednesday : 2013.03.13

• 08:45 Bill: cleanup seems to be running very slowly. It's pantasks_server and pcontrol are cpu hogs. Set to stop in preparation for restarting it.
• 08:55 Bill: set staticsky to off in the stack pantasks. It is currently working on skycells in the galactic plane and is running into memory problems on the compute2 nodes. After it settles out will start up the deepstack pantasks for a couple of days.
• 10:50 CZW: stopped cleanup at Serge's request. This should allow ippdb02 to catch up within 72000 seconds.
• 11:15 CZW: restarted stack and stdscience to allow Bill to fork staticsky processing back to the deepstack pantasks. The restart was needed to ensure the compute3 resources are allocated appropriately.

Thursday : 2013.03.14

Bill is czar today

• 19:07 Bill: stdscience pcontrol is spinning. Setting stdscience to stop in preparation for restart.
• 19:47 restarted stdscience, registration, and summitcopy

Friday : 2013.03.15

Bill is czar today

• 09:49 ipp020 has been down for a couple of hours. Power cycled it. Beginning of crash dump was

  <Mar/15 08:35 am>[3690682.353980] ggeneral protection fault: 0000 [#1] SMP 
  <Mar/15 08:35 am>[3690682.354269] last sysfs file: /sys/class/i2c-adapter/i2c-0/name
  

• 12:20 Since staticsky has moved out of the galactic plane, doubled the number of compute3 hosts working on staticsky in deepstack

Saturday : 2013.03.16

Sunday : 2013.03.17