(Up to PS1 IPP Czar Logs)

Monday : 2012.07.16

Bill is czar today.

• 10:10 stdscience was stuck. There are several thousand chip runs outstanding but no jobs are getting queued. Czartool indicates that chip processing stopped making progress around 20:15 last night while outstanding warps got finished just after midnight. The book chipPendingImfile had lots of entries in init state but nothing was getting run. I restarted stdscience.
• 13:40 Chris discovered that ippc03 was out of space in /tmp which was causing nebulous requests to fail. This undoubtedly contributed to the processing failures. Bill deleted the log files and restarted the apache servers.
• 13:55 chip reverts are off because we have a number of exposures that are faulted repeatedly due to incorrect burntool state. Chris is working on fixing the burntool states.
• 15:18 I see that chip reverts have been turned back on.

Tuesday : 2012.07.17

Bill is czar again today

• 10:18 LAP is done through warp except for 4 exposures that have chips with missing burntool tables and one warp process whose perl script got stuck trying to check files on one of the stsci nodes. Killed that process and the warp completed. Also had to rerun chip 493934 XY55 because the output variance image was corrupt.
• 10:45 Used Chris's do_burntool_single.pl script to recreate the missing burntool tables.
• 11:15 ipp008 has some chip processes that have been stuck. Gene and I tried to fix the problem to no avail. Rebooted the system.
• 11:23 restarted stdscience
• 13:23 restored a couple of "lost" raw imfiles with previously unknown copies that Serge was able to find with his new databases. More info later.

Heather is the afternoon czar

Wednesday : 2012.07.18

Mark is czar

• 02:30 fixed ppStack bug, turning LAP stacks back on for another test sample. about 900/3500 likely had this problem in the earlier run of stack.
• 03:10 the 140 extra stack processing is hitting the stsci machines fairly hard, turning off 1x stare+2x compute+1x compute2 (~35) to start with. still causing some large spike, taking out 1x stare+1x compute2+1x wave4 more (~27).
• 09:40 czar rate plot indicates ~300 stacks/hr with ~82 nodes. stsci nodes still getting spiked (nfs) but have handled for past few hrs.
• 11:00 adding back 1x compute2+1x stare+1x wave4 for 103 nodes, stsci node spikes are broader now (time) -- monitoring -- seems okay and reaching >400/hr
• 11:30 ganglia reports ipp008 all memory swapped but isnt. gmond restart worked for a little bit but back to saying all memory swapped. keep eye on.
• 12:00 ecliptic.rp chip faults often NON-EXISTANT, but are being found with the new deneb-locate.py command --

  	o5362g0549o 	XY36 	181808 	488192 
  neb://ipp030.0/gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits
        0                     NON-EXISTANT file:///data/ippb02.1/nebulous/d1/13/917781576.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
        0                     NON-EXISTANT file:///data/ippb02.0/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
  
  -- rogue files?
  /home/panstarrs/ipp/local/bin/deneb-locate.py gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits
  /data/ippb02.2/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits -- seems ok
  /data/ipp024.0/nebulous/d1/13/319361864.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits -- 0 size
  
  -- copy over to non-existant file location
  cp /data/ippb02.2/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits /data/ippb02.1/nebulous/d1/13/917781576.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
  cp /data/ippb02.2/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits /data/ippb02.0/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
  
  -- want to replicate to main machines so first instance is not looked for on ippb machines
  --> could also replicate all off of ippb machines since overly full -- if not use volume then likely puts to stsci randomly and don't want
  neb-replicate neb://ipp030.0/gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits --volume ipp030.0
  neb-replicate neb://ipp030.0/gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits --volume ipp031.0
  
  neb-cull neb://ipp030.0/gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits --volume ippb02.0
  neb-cull neb://ipp030.0/gpc1/20100615/o5362g0549o/o5362g0549o.ota36.fits --volume ippb02.1
  
  -- once reverted and okay, can rm rogue files
  rm /data/ippb02.2/nebulous/d1/13/917799349.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
  rm /data/ipp024.0/nebulous/d1/13/319361864.gpc1:20100615:o5362g0549o:o5362g0549o.ota36.fits
  
  

• 12:10 ecliptic.rp chip faults also from missing/lost on ipp064.0X

  	o5501g0222o 	XY26 	248870 	488662 
  neb:///gpc1/20101101/o5501g0222o/o5501g0222o.ota26.fits 
  --> no instances
  whichnode neb:///gpc1/20101101/o5501g0222o/o5501g0222o.ota26.fits
  ipp064.0X not available
  ipp064.0X not available
  
  --
  
  

• 12:30 adding back 1x compute+1x compute2 for 124 nodes in stack pantasks -- gets the stsci nodes to load spike to >100 with nfs.. but handling so far..
• 12:50 pile up of ecliptic.rp in camera stage, ~15 runs >20ks. killing and reverting to see if clear ok -- when reverted still stall -- need to be looked into.
• 13:58 CZW: Added SAS_v8 to stdscience. Defined LAP queue for Kepler field (based on ​http://kepler.nasa.gov/Science/about/targetFieldOfView/, defined as (227.5,36)->(302.5,52.5)) and set it as the "current.queue". This means that as the 09:00hr slice finishes, it will launch Kepler fields. I've also defined a 10:00hr slice, so we can switch to that when Kepler is fully queued.
• 16:00 short testing single stage processing for chip/camera/warp/stack, warp is the primary culprit for harassing the stsci machines.
• 18:00 running SAS chips/camera only for full rate (running set.poll 550)
• 21:00 SAS chips finished, turning on warps to run slowly overnight and monitor rate. LAP chips/camera also processing since should cause no extra slow down.
• 23:50 LAP stacks turned back on once SAS warp rate seen to turn over, adding stacks not appear to decrease rate significantly.

Thursday : 2012.07.19

Mark is czar

• 10:00 processing stopped for Gavin to raise the number of nfsd instances on the stsci machines from 128 to 256. restarted processing with similar chip+cam+warp+stack as before (diff.off) to monitor rate. seems has the ability to push load up to 200 now but the system is responsive and just waiting on the many IO to the disk, same wait/load problem exists. *
• 12:30 removing ecliptic.rp from stdscience and placing back in its own pantasks at a lower rate.
  • turning up processing on LAP diffs (all to run in stdscience, set.poll 500 -2x compute3 -2x stare) and SAS+LAP stacking (adding 1x compute3, -1x stare overloaded, +1x wave3, -1x compute2 to accomodate ecliptic pantasks)

Friday : 2012.07.20

Saturday : 2012.07.21

Sunday : 2012.07.22