PS1 IPP Czar Logs for the week 2013.07.29 - 2013.08.04

(Up to PS1 IPP Czar Logs)

Monday : 2013.07.29

mark is czar

• 08:30 MEH: doing regular restart of stdsci (turndown in rate)
• 08:50 MEH: update appears down, restarting. adding 1x compute3 (taking from stdsci)
  • might as well restart other main running ones like summitcopy, registration.
• 09:15 MEH: and restarting pstamp
• 10:10 MEH: in preparation for possible power issues with Flossie, stopping all processing and shutting down compute2+3 nodes (and stare which Gene will specifically take care of for dvo/ipptopsps issues). also ippc30 still PSS pseudo-datanode so leave up as well
• 11:15 MEH: compute3 ippc63--ippc31 shutdown started

• 11:45 MEH compute2 ippc29--ippc20 shutdown started

• 12:15 MEH compute2 ippc10,c14,c15,c16 started

• 19:00 Gene powered down all systems

Tuesday : 2013.07.30

mark is czar

• 07:30 MEH: Gavin proceeding with long processes of restarting all machines
• 09:50 MEH: Gene has made the shift of the /data/ipp031.0, /data/ipp032.0 locations to stsci03.0, stwci00.0 respectively with lower level symlinks under nebulous across all stsci machines
• 10:00 MEH: most machines up, doing secondary check of cpu, memory etc not MIA as we have occasionally found in past (also datetime correct)
  • ipp061 refound its lost 8G RAM from end of March
  • ipp060 lost 8G RAM.. doing shutdown and powerdown/up restart -- 8G back
• 10:10 Gavin reports

  ippc02 - down due to audible alarm
          (MHPCC are escorting guests and requested we disable it)
  
  ippc53 - checking on it, console is blank but I can see outlet
          is providing 263W.
  
  ipp046 - not powering up after power outlet cycle.
          (suspect BIOS settings not being saved.
          time for button battery replacement..)
  

  • Haydn checking on ippdb04 at ATRC -- up an running now, ganglia thinks it is down still
• 10:30 MEH: ippdb00,02 mysql not running so need to start -- see Processing
  • apache already running on ippc01-09 (taking ippc02 out of list in ~ipp/.tcshrc)
  • nebdiskd started by ipp@ippdb00
  • neb-host ipp046 down
  • ippc02,c53,046 commented out in ~ippconfig/pantasks_hosts.input
• 11:00 MEH: restarted czar screens and czarpoll on ippc11
• 11:10 MEH: still scanning nfs mounts across all system
• 11:20 MEH: ipp028,033,036 commonly missing, after a couple nfs restarts and minute they are exporting okay -- sudo /etc/init.d/nfs restart

  -- first restart reported error
  exportfs: Warning: /export/ipp028.0 does not support NFS export.
  exportfs: Warning: /export does not support NFS export.  
  

• 11:40 MEH: neb-host repair for ipp033, 041 (red and full) so if cleanup frees up space still allows space for dvo work (as with others in that wave)
• 12:00 MEH: ipp046 back up, so back to neb-host repair. ippc02 also back up so back into the .tcshrc nebservers. -- both added back into ~ippconfig/pantasks_hosts.input
• 12:05 MEH: stdsci pantasks started, will start PSS set next
• 12:40 MEH: all pantasks started and running again
• 12:50 MEH: ippc02 taking out of processing until alarm is tracked down, but leave as nebserver. ippc53 rebuilding software raid so leaving out as well.
• 13:32 Bill: ran the commented out survey.add.relexp and survey.add.relstack commands in stdscience. I can find nothing wrong with the setup. Changed input file to uncomment them for next time stdscience is restarted.
• 15:12 Bill: queued r band M31 exposures from first half 2010-09 to be processed
• 15:20 queued m31.rp distRuns that MPG has downloaded to be cleaned.

Wednesday : 2013.07.31

• 13:25 MEH: looks like LAP stacks won't be made for quite some some, reallocating compute3 power back to stdsci.. stdsci also in desperate need of regular restart.. doing now
  • also adding back in ippc53 taken out yesterday for software raid issue

Thursday : 2013.08.01

• 09:30 EAM : ipphosts was overloading the stsci machines : set.host.by.skycell was resulting in hosts of 'stsci07.1', which does not exist. thus the rules for runaway overload of a machine are not respected. i've fixed set.host.by.skycell to strip off the .1 leaving just the real host name.
• 09:31 EAM : ipp025 is hung up on nsf, rebooting (everything currently stopped)
• 10:20 EAM : the ppSkycell jobs (skycell_jpeg.pl) were overloading the stsci nodes (or contributing to the load). I've deactivated them for now.
• 13:00 MEH: in the meantime, stealing 1x compute3 from unused stack pantasks for test stacks/diffims in local pantasks
• 15:00 MEH: to make last nights MD stacks need to add date back ns.add.date 2013-08-01
• 15:40 MEH: stsci still getting hit hard even with diluting warps by having chips run (70/10 chip/warp %..), stopping cleanup to see if eases things a bit
  • same goes for distribution
• 16:20 MEH: put stsci00 into repair, 2/3 disks are red. has load >200 for past 30 min..
• 16:40 MEH: cleanup+dist stopped not helping, putting stdsci to low poll and will raise until things break..
• 17:00 MEH: stsci00 load finally <100 after ~hour, distribution back on and ok.
  • setting stdsci poll 100, after a bit jumps to maintained loads of ~100
• 17:45 MEH: enough of the wack-a-mole... try shutdown of mysql on stsci00 -- load dropping and back to normal levels with full processing and neb-host up again.
• 18:15 MEH: finally all but one edge skycell are finished for the MD stacks.. tweak_ssdiff to get the diffims queued before nightly starts.
• 18:30 MEH: noticed with chip.off, system not fully loaded even though plenty of diffs and warps to process.. something is keeping them from fully load, poll rate?

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD