(Up to PS1 IPP Czar Logs)

Monday : 2012-09-17

Bill is czar today.

• 09:00 LAP processing is stuck due to a diff cleanup issue Chris will repair.
• 09:56 stopping processing in order to restart all pantasks setting turning ipp037 on as a compute node.
• 10:30 LAP is moving along nicely now
• 10:38 repaired a number of missing files (found with deneb-locate). Two were actually lost for good: o5364g0119o.ota65.fits and o5361g0129o.ota65.fits
• 13:11 ipp020 has lost communication with the virtual /data/ipp037.0 (stsci00) shut it down then power cycled after the usual hang in "unmount nfs file systems" Because I'm lazy today I'm removing the host from processing lists.
• 15:07 setting lap.off for a bit so we can debug the state of the confused state czartool's chip data without the counts changing out from under us.
• 15:40 ipp028 has gone out to lunch. Lots of nfs errors in dmesg. Something about general protection fault as well but that did not show up in the console log. Power cycled it.
• 15:55 turned lap.on but left lap.cleanup.off for a little while longer
• 17:00 ipp028 is not visible in nebulous but it seems to be up. Setting it to repair state for now.
• 17:14 Ok I'm convinced that czartool is just getting confused by the goto_cleaned rate. Turned lap.cleanup.on

Tuesday : 2012-09-18

Bill is czar again today

• 06:40 Processing rate dropped through the night to ~60 per hour from 100 per hour. Restarted stdscience. ipp020 are in host lists but in repair mode for nebulous until the sun is higher in the sky.
• 06:47 reduced nhosts in stdscience from 6 to 3 on ipp020 (24GB) and from 4 to 2 on ipp028 (16MB). Set them to up in nebulous
• 09:33 at Mark's suggestion added compute3 hosts to stdscience since deepstack isn't using them
• 10:50 MEH: stopping all pantasks to rebuild ippTools, restarting stdscience to make the MD label change to MD09.refstack. manually setting ipp020, ipp028 back to lower level as above and adding back in extra compute3.
  • tweaking ssdiffs to try running before noon (test if difftool+refstack label change working), new MD09.refstack.20120831+staticsky run to distribution.
  • 7 "new" SSdiffs for skycell.082 since previous refstack had quality issue with it. seems to work, need to watch with new nightlyscience for over-queue in future.

Wednesday : 2012-09-19

heather is czar

• 11:36am restarting stdsci

Thursday : 2012-09-20

Mark is Czar

• 08:40 MEH: probably won't help, but doing stdscience restart. few chips+warps running for >40ks
  • also throwing in an extra compute3 to stdscience since deepstack not running
• 09:25 EAM : taking ipp037 out of processing -- as part of ongoing testing, I am going to set it to be 'on' in nebulous, and off in processing. but first, I need to ensure the rsyncs are still up-to-date
• 10:10 MEH: still no LAP stacks to do. until stacks queued, going to micromanage stdscience and try pressing more compute nodes into service..
  • (10:25) didn't take too long to get stacks.. will test again later if happens to look at if any throughput increase or if just more faults.
• 11:40 MEH: noticing compute2 often going into swap hell. pswarp 20% of 24GB on ippc21 for example, so removing 1xcompute2 (~10 nodes) set from stdscience.
  • even with 1xcompute3 to balance out other moved nodes to keep things smoothed out, still only ~50/hr in stdscience in GP
• 12:40 MEH: stopping and shutting down all pantasks/processing
  • need to reboot ipp046 and ipp050 due to NFS fault and not being able to access locally exported disk -- ipp046 not coming back up, is it one with BIOS set to not reboot after powercycle??
  • Gene/Gavin work with ipp037 -- will put neb-host up for disks but keep out of processing. can turn on cleanup for the r/w active test.

Friday : 2012-09-21

Saturday : 2012-09-22

Sunday : 2012-09-23